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Dear Maine Educator: 

In the spring of 2003, my Policy and Technical Advisory Committees for the 
Comprehensive Assessment System recommended that the Department conduct a Local 
Assessment System Implementation Study to see how a diverse sample of school administrative 
units (SAUs) were responding to the demands of Local Assessment System (LAS) development. 
Led by a strong research group, the design was completed and SAUs were selected. 

I am pleased to provide you with a copy of the Local Assessment System Implementation 
Study (2003-2004): Technical Report that includes the results of our 2003-2004 study that 
examined the actions taken by these 1 8 school districts in response to the challenges of 
developing a LAS using the principles and criteria set forth in the Local Assessment System 
(LAS) Guide distributed to school districts in June, 2003. 

The report is organized into three parts: Part I is the Introduction, Part II the Results, and 
Part 3 the Implications. The report details the rationale for our study, the methodology we used, 
an analysis of the results, and the Maine Department of Education (MDOE) actions based on the 
implications of the study. 

The results are intended to enhance our understanding by examining the actual experiences 
of school administrative units as they engage in developing and implementing Local Assessment 
Systems. As an outcome, the MDOE will be better prepared to assist local school districts with 
this work. The report concludes with the implications of the study and the actions that the 
MDOE will take to respond to these implications. 

Special thanks go the 1 8 school administrative units whose teams of educators 
contributed significantly to our collective learning about LAS development and implementation, 
and to the research team, led by Dr. Ted Coladarci, who guided the work. 



Sincerely, 




Susan A. Gendron, Commissioner 
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PARTI: INTRODUCTION AND METHOD 



When the Maine State Legislature established the Learning Results in 1996, it stipulated 
that student achievement of these standards was to be measured by a combination of state and 
local assessments. The latter component gave rise to the notion of “local assessment system” 
(LAS), which, the Maine Educational Assessment notwithstanding, is the dominant approach in 
Maine for assessing student achievement in the present climate of standards-based education. In 
such an approach, the emphasis is on locally developed (or locally selected) assessments that are 
curriculum-embedded and provide students with multiple opportunities to demonstrate 
achievement. Further, these assessments are to be of sufficient quality to yield results that can 
inform teaching and learning, hold school administrative units (SAUs) accountable, and certify 
students’ achievement of the Learning Results} 

Although not unique, Maine is one of the few states (Nebraska also comes to mind) to 
hold local assessment in such high regard. But the corresponding challenge is that there are no 
proven models for carrying out this work. The Maine Department of Education (MDOE) has 
provided guidance along the way, as can be seen in such documents as Measured Measures: 
Technical Considerations for Developing a Loccd Assessment System , LAS Guide: Principles 
and Criteria for the Adoption of Loccd Assessment Systems, and most recently, Considering 
Consistency: Conceptual and Procedural Guidance for Reliability in a Local Assessment 



1 See Chapter 127, Section 4, of the Department’s regulations governing Instructional Programs, Assessment, and 
Diploma Requirements (ftp://ftp.state.me.us/pub/sos/cec/rcn/apa/05/071/071cl27.doc) 

2 Coladarci, T., Johnson, J. L., Beaudry, J., Cormier, M., Ervin, R., Rosenblum, J. M., & Silvernail, D. L. (2000). 
Measured Measures: Technical Considerations for Developing a Local Assessment System. Augusta, ME: Maine 
Department of Education, (http://www.state.me.us/education/g2000/measured.pdf) 

3 Maine Department of Education (2003). LAS Guide: Principles and Criteria for the Adoption of Local 
Assessment Systems. Augusta: Author, (ftp://ftp.state.me.us/pub/sos/cec/rcn/apa/05/071/071cl27.doc) 
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System . 4 Nevertheless, the nature and demands of LAS work take on increasingly greater clarity 
as SAUs engage in this important effort. Although it would be inaccurate to say that we are 
“making it up as we go along,” it is true that we are learning more and more about this 
undertaking as we proceed. As acknowledged in the introduction of the LAS Guide, it therefore 
is important that we have “periodic pauses for reflection and adjustment” (p. 1). The LAS 
implementation study is just such a pause. 

The recommendation that the MDOE conduct a LAS implementation study was made in 
spring 2003 by the Policy Advisory Committee and Technical Advisory Committee, which held 
a joint meeting to discuss the principles and criteria that ultimately were incorporated into the 
LAS Guide. Why conduct such a study? In short, it is to see how a diverse sample of SAUs are 
responding to the challenges of LAS development. What aspects of LAS development seem to 
be going well for these SAUs? What “boosters” are perceived — i.e., those circumstances, 
resources, or events that are believed to facilitate the LAS development process? What 
“barriers” are perceived — i.e., those circumstances, resources, or events that are believed to 
impede progress in LAS development? By understanding the actual experiences of SAUs as 
they engage in the important work of LAS development, the MDOE is better positioned to help 
all SAUs build capacity in this regard and, ultimately, to help SAUs move toward the realization 
of the Learning Results. 

LASIS research group 

The MDOE assembled a research group in July 2003 to design the local assessment 
system implementation study, or LASIS, as it quickly came to be known. The initial LASIS 
group comprised Patrick Phillips, Pam Rolfe, and Tom Keller (MDOE); Jill Rosenblum (Maine 

4 Maine Department of Education (2003). Considering Consistency: Conceptual and Procedural Guidance for 
Reliability in a Local Assessment System. Augusta: Author. 
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Mathematics and Science Alliance); Walt Harris and Lori Smith (UMaine Center for Research 
and Evaluation); Gary MacDonald (New Suncook School); Jeff Beaudry (University of Southern 
Maine); Ted Coladarci (UMaine); and Brian Gong and Charlie DePascale (Center for 
Assessment). The few modifications to this group were that the Center for Assessment was 
involved only in the design phase of LASIS and three research assistants were later hired for data 
collection purposes (Sara Bryant, Wayne Garthwait, and Owen Maurais). 

The 2003-2004 LASIS design (described below) reflected a modest budget, the desire to 
limit this study of preliminary LAS development to a single school year, and a resolve to not 
impose on participating SAUs any more than necessary. 

Participating SAUs 

The MDOE selected 18 SAUs from a pool of (mostly) volunteers. There were two 
SAUs from each of the nine superintendent regions. By having each region represented, a 
certain degree of representation also was achieved with respect to system size, demographic 
characteristics, and other correlates of location. Whenever possible, SAUs within a region were 
selected that differed in LAS implementation status so that, across all LASIS sites, there would 
be variability in this regard. Such variability was necessary in order to maximize the opportunity 
to uncover perceived boosters and barriers to LAS development. 

LASIS design components 

The LASIS design components are described below with a brief rationale for each. 
Instruments are presented in appendices. 

Informational meeting and initial survey. An informational meeting was held in 
Waterville on November 7, 2003, for all 18 LASIS sites. Each site was represented by a team 
composed of those individuals most knowledgeable about and involved in the LAS work in the 
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SAU. After the MDOE made opening comments about LASIS and provided an opportunity for 
participants to ask preliminary questions, each site team, at its own table, discussed and 
completed an initial survey that was structured around the criteria and rules embodied in the LAS 
Guide. Presented in Appendix A, this initial survey was designed to provide (through selected- 
response items) a snapshot of each site’s LAS implementation status at that point in time as well 
as to solicit (through open-response items) participants’ perceptions regarding factors that either 
facilitated or hindered LAS development in their SAU. Equally important, the initial survey 
provided a structured opportunity for each site — perhaps for the first time — to collectively, 
deliberatively, and systematically reflect on where they were in LAS development. 

A team member at each table recorded responses to both parts of the survey on a laptop 
computer. (A disk with this information was later given to LASIS researchers.) These responses 
also were transferred to poster notes and displayed for all sites to view in a “gallery walk.” 
Finally, this day-long event also provided the opportunity for LASIS participants to form mixed- 
site discussion groups to leam about the experiences of other SAUs regarding LAS development. 

Phone interviews and site visits. An extended phone interview was scheduled with a key 
LAS individual in each participating site to explore more fully what was anticipated from the 
initial survey as well as to inquire about issues that a paper-and-pencil instrument simply cannot 
accommodate. Most phone interviews were held between December 2003 and January 2004. 
(The phone interview protocol is presented in Appendix B.) A site visit was conducted both as a 
follow-up to the initial survey and to probe and clarify what was anticipated from the phone 
interviews. Conducted during the first several months of 2004, each site visit involved meeting 
with those individuals closely involved with LAS development in the respective SAU. (The site 
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visit protocol is presented in Appendix C.) The three research assistants conducted all phone 
interviews and site visits. 

Peer review of LAS documentation. Site teams were brought together on May 17-18 in 
Waterville to participate in a review of each other’s LAS documentation. Each site’s LAS 
materials were examined 2-3 times, each time by a different mixed-site group of individuals. 
The reviewers’ charge was to detennine the degree to which there was evidence in the SAU’s 
LAS materials that progress had been made in enacting the various principles and criteria 
embodied in the LAS Guide. This component of LASIS had several purposes. Lirst, each site 
would profit from the task of taking inventory of what it had available by way of LAS 
documentation and, in turn, organizing this information for peer review. (See Appendix D for 
the LAS checklist that guided this effort.) Second, each site would benefit from the peer review 
itself: an appraisal, by one’s peers and based on submitted evidence, of the progress the SAU 
had made in enacting the LAS Guide. (The peer review instrument is presented in Appendix E.) 
Third, participants would be equipped with a helpful tool for self-assessment as they continue to 
develop their LAS. Linally, the resulting information from systematic review would be an 
additional source of data for gauging SAU accomplishments. 

June 2004 survey. In June 2004, all SAUs in Maine — whether LASIS participants or 
not — received a comprehensive survey that assessed their progress in, and beliefs about, LAS 
development. 5 The results of this survey provide the most current portrait of implementation 
status among LASIS participants. (The June 2004 survey can be found in Appendix G.) 
Although similar in purpose to a survey administered in June 2003, the earlier survey was 
modified appreciably to produce the improved June 2004 survey. With a few exceptions, 
consequently, the two surveys do not pennit meaningful comparisons. 

5 The UMaine Center for Research and Evaluation will present the state -wide results in a separate report. 
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PART II: RESULTS 

LASIS results are presented below in four sections: results from (a) the initial survey (b) 
the phone interviews and site visits, (c) the peer review of LAS documentation, and (d) the June 
2004 survey. 



THE INITIAL SURVEY: NOVEMBER 7, 2003 
All 18 LASIS sites were brought together in Waterville on November 7, 2003 for the 
purpose of sharing their experiences regarding LAS development and implementation. 
Participants at the meeting were invited to complete a two-part survey of the current status of 
their local assessment system and the perceived “boosters” (facilitating factors) and “barriers” 
(challenges and obstacles) with respect to LAS development. Each site team, talking through the 
issues and considering the status of their LAS, collectively completed a single survey for their 
SAU. Presented in Appendix A, this initial survey was designed to provide a snapshot of each 
site’s LAS implementation experience at that point in time. 

Parti 

Part I of the survey comprised selected-response items structured around the five criteria 
in the LAS Guide. Additionally, there was a more general item in Part 1 that inquired about the 
overall status of LAS development in each of the five content areas and at three grade spans. 
Reflecting on their SAU’s understanding, support, implementation, and evidence, respondents 
indicated whether they were “exploring,” “transitioning (early),” “transitioning (more 
advanced),” or “transforming.” These terms were defined for participants as follows: 

• Exploring - Becoming acquainted with the ideas. Still building support for an LAS 
within the district and schools. Not very familiar with the LAS Guide. Still in the early 
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planning stages. Documentation may be available to some, but not in a consistent form 
or quality for public reporting or peer review. 

• Transitioning ( early) - Have good beginning with assessment literacy and have good 
capacity (many people) to build towards a comprehensive LAS. Key persons in district 
and schools have a working knowledge of most of the LAS Guide. Have strong support 
for the LAS in key parts of the district/schools. Have several parts of an LAS in place, 
but we are working on getting alignment and agreement across schools, grades, teachers, 
and/or content areas. Documenting our process as we go. 

• Transitioning (more advanced ) - Have district LAS policies and procedures in place. 
Many people (administrators, faculty, students, parents) understand and implement 
system well. The value of the LAS is reflected in such things as regular reporting to 
policy makers and stakeholders, integrated use in planning, and in discussions of 
allocation of resources. Have established processes to review the system components, 
and have documented evidence of how system meets principles of coherence, sufficiency, 
and fairness. 

• Transforming - Many teachers can extend the LAS beyond district and state 
requirements. District has fine-tuned LAS requirements to fit its individual 
circumstances, while meeting core principles of coherence, sufficiency, and fairness. 

LAS results are incorporated regularly into district and school decisions as valuable data. 
District has a thorough process and regularly reviews evidence to certify and improve 
LAS. 

Analysis of these data showed that participating sites had devoted most of their time and 
energy to tackling the first two criteria (assessment types and comparability), with considerably 
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less time going to the last three criteria (replacement, standard setting, and public reporting). 

This is understandable, of course, given the logical/temporal progression of LAS development 
and implementation. 

Analyses of the overall status of the LAS in each of the five content areas revealed two 
general themes. First, LAS development tended to be further along in English/language arts, 
mathematics, and science than in the remaining content areas. This, too, is understandable, given 
the initial timeline for LAS development and implementation specified in Chapter 127. The 
second theme is that more progress had been made in LAS development at the secondary level 
than at the elementary or middle level. Given the accountability pressures regarding high school 
graduation, this is understandable as well. Tables 1-5 below present the results of this 
overarching item. 6 



Table 1. Current Phase of LAS development in English/Language Arts 





Grades K-4 


Grades 5-8 


Grades 9-12 


n 


% 


n 


% 


n 


% 


Exploring 


3 


16.7% 


3 


16.7% 


2 


11.1% 


Early Transitioning 


10 


55.6% 


13 


72.2% 


10 


55.6% 


Transitioning 


2 


11.1% 


2 


11.1% 


2 


11.1% 


Advanced Transitioning 


2 


11.1% 


0 


.0% 


2 


11.1% 


Transforming 


1 


5.6% 


0 


.0% 


2 


11.1% 



6 Because several sites indicated their current phase of development was located somewhere between early 
transitioning and advanced transitioning, a fifth category transitioning was created. 
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Table 2 . Current Phase of LAS development in Mathematics 





Grades K-4 


Grades 5-8 


Grades 9-12 




n 


% 


n 


% 


n 


% 


Exploring 


2 


11.1% 


5 


27.8% 


3 


16.7% 


Early Transitioning 


10 


55.6% 


8 


44.4% 


9 


50.0% 


Transitioning 


1 


5.6% 


2 


11.1% 


0 


.0% 


Advanced Transitioning 


4 


22.2% 


3 


16.7% 


4 


22.2% 


Transforming 


1 


5.6% 


0 


.0% 


2 


11.1% 



Table 3. Current Phase of LAS development in Science and Technology 





Grades K-4 


Grades 5-8 


Grades 9-12 


n 


% 


n 


% 


n 


% 


Exploring 


9 


52.9% 


7 


41.2% 


3 


17.6% 


Early Transitioning 


5 


29.4% 


9 


52.9% 


10 


58.8% 


Transitioning 


2 


11.8% 


1 


5.9% 


1 


5.9% 


Advanced Transitioning 


1 


5.9% 


0 


.0% 


2 


11.8% 


Transforming 


0 


.0% 


0 


.0% 


1 


5.9% 



Table 4. Current Phase of LAS development in Social Studies 





Grades K-4 


Grades 5-8 


Grades 9-12 


n 


% 


n 


% 


n 


% 


Exploring 


9 


50.0% 


7 


38.9% 


2 


11.1% 


Early Transitioning 


8 


44.4% 


11 


61.1% 


12 


66.7% 


Transitioning 


0 


.0% 


0 


.0% 


1 


5.6% 


Advanced Transitioning 


1 


5.6% 


0 


.0% 


2 


11.1% 


Transforming 


0 


.0% 


0 


.0% 


1 


5.6% 



Table 5. Current Phase of LAS development in Health and Physical Education 





Grades K-4 


Grades 5-8 


Grades 9-12 


n 


% 


n 


% 


n 


% 


Exploring 


10 


55.6% 


10 


55.6% 


8 


44.4% 


Early Transitioning 


7 


38.9% 


6 


33.3% 


7 


38.9% 


Advanced Transitioning 


1 


5.6% 


2 


11.1% 


3 


16.7% 
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Part 2 

Part 2 of the survey comprised open-response questions. In large part, these questions 
elicited perceptions regarding boosters and barriers with respect to LAS development. In broad 
brushstrokes, responses to these questions are summarized below. 

Boosters. The importance of having strong leadership frequently was cited as a 
facilitating factor, as was the dedication and commitment of staff. Of course, these factors would 
be inconsequential without the provision of time (e.g., for professional development and for 
doing the LAS work itself), which also surfaced as an important booster. Other facilitating 
factors concerned the ability to pursue and obtain grants to support LAS work, the creative use of 
existing resources, and the exploitation of assistance opportunities provided by the MDOE. 

Barriers. Not surprisingly, some of the perceived impediments to LAS development 
reflected the absence of the boosters just noted. This is particularly true in regard to leadership 
(e.g., administrative turnover) and staff commitment (e.g., philosophical differences, resistance 
to change). Too little time, inadequate resources, and insufficient capacity-building 
opportunities also were mentioned as limiting progress in LAS development. Another perceived 
barrier is the late arrival of the LAS Guide and related information regarding LAS development 
and implementation. Finally, there was the sense of simply feeling overwhelmed by the 
magnitude and timeline of this task, particularly in light of the additional accountability pressures 
from No Child Left Behind. 

PHONE INTERVIEWS AND SITE VISITS 

As described above, an extended phone interview was scheduled with a key LAS 
individual in each participating site, which was followed by a site visit that involved a number of 
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individuals involved in the LAS effort. (Protocols for the phone interviews and site visits can be 
found in Appendices B and C, respectively.) This section begins with a description of the 
various ways in which SAUs organize themselves to accomplish the LAS work. Following this 
is an accounting of the perceived “boosters” and “barriers” regarding this work. This section 
concludes with a distillation of findings from the phone interviews and site visits. 

How School Administrative Units Approach LAS Work 

Structures for accomplishing LAS work range from informal to fonnal: from teachers 
functioning with little or no administrative guidance to interlocking committees whose 
responsibilities are clearly defined in organizational charts. The most developed structures 
feature regular administrator participation with teacher leaders. Usually, the clearly defined and 
formal structures were not specifically designed for LAS work, but, rather, are the legacy of 
SAU efforts to effect other systemic change (e.g., curriculum mapping to implement Learning 
Results). It is not coincidence that those SAUs with strong traditions of curriculum analysis, 
collegiality, aggressive grant writing, and professional development tend to be the SAUs whose 
structures for LAS work are most sophisticated. 

Administrative Support 

Table 6 reports the results regarding administrative involvement in LAS work, where one 
can see that over half of the SAUs indicate that a curriculum coordinator or director of 
instruction was directly involved in LAS work. These reporting categories are not mutually 
exclusive. That is, an SAU that reports active participation by the superintendent may have a 
principal or curriculum coordinator who is involved as well. 

Although data were not available from all the SAUs, four SAUs that estimated a 
percentage of administrator’s time spent on LAS-related work reported that curriculum 
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coordinators averaged between 20% to 30% of their time and principals from 5% to 20%. 
Estimates from one SAU placed principal involvement at a few hours per week. 



Table 6. Administrator Involvement in LAS Work 



Direct involvement by: 


N of SAUs 


% of SAUs 


Superintendent 


2 


11 


Assistant Superintendent 


4 


22 


Curriculum Coordinator/Director of Instruction 


10 


55 


Principal(s) 


9 


50 


Little or no direct involvement of any administrator 


2 


11 


Note: N= 18. Direct involvement in a given SAU may include more than one position. 



Data gathered in both telephone interviews and onsite interviews indicate that an 
administrator usually coordinates the overall effort through principals, committees, and/or 
teacher leaders. More fonnal structures may involve a combination of all. Usually teacher 
leaders, often assisted by administrators on release days or in-service days, function as trainers 
and disseminators to the rest of the faculty. The teacher leaders attend MDOE training 
workshops, collaborate with each other to develop assessments, sit on committees, and in some 
cases, meet regularly with their counterparts from other SAUs. Teacher leaders are typically 
organized by grade cluster in the earlier grades and by discipline in middle and high school. 
Finally, almost without exception, teacher leaders are self-selected. 

The following statements about organizational structure for accomplishing LAS-related 
work characterize most SAUs that appear to be further along in LAS development: 

• The organizational structure predates LAS, but has been adapted to develop an 
LAS. 
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• The organizational structure includes administrators, a curriculum specialist, and 
teacher leaders. 

• The organizational structure features central coordination. 

• Roles, goals, and responsibilities, including communication, are clearly defined and 
can be represented graphically. 

In one SAU that appears to have made significant progress in developing its LAS, a 
“curriculum coordinating council” had been previously established to oversee K-12 curriculum, 
assessment, and instruction. It developed and monitored district timelines, analysis of data, 
NCLB, and program/textbook approval. Below the Council are K-12 vertical teams, each co- 
chaired by a building administrator and an “instructional teacher leader.” There are, on average, 
ten people on each vertical team, including teachers from all buildings and grade spans, 
administrators, and, in some cases, a parent. These teams’ responsibilities include aligning 
curriculum, developing and inventorying common assessments and pilot assessments, and 
communicating to the council and staff. Vertical teams feature subcommittees organized on a 
building level. Each building, with one exception, features seven stipended instructional teacher 
leaders who report out and facilitate the work. 

Given the various components of an LAS as outlined in the LAS Guide, one would expect 
that the tasks associated with LAS development would be shared by administrators, policy 
makers, curriculum specialists, and teachers. Clearly, even the most accomplished and energetic 
teacher leader cannot develop on his or her own a system that includes, for example, policy 
development, inservice day planning, and resource allocation. Thus, in those SAUs where there 
is little or no administrative involvement in the process, work may not have been able to advance 
beyond identifying pilot assessments. In those SAUs operating within a more organized and 
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coherent structure, most tasks and/or sample assessments appear to originate from the 
committees of administrators and teacher leaders who are spearheading the work. In some 
instances, these are provided as templates for individual teacher work. Sometimes, they are 
offered as examples for the teachers to analyze and emulate at the discretion of the individual. To 
the greatest extent possible, groups of teachers are given latitude to select and to develop their 
own assessments, provided such materials comply with the LAS Guide. Even in less organized 
SAUs, teachers do not report working in isolation from their peers. 

Provision of Time 

When is all this work being done? How do teachers and administrators pry additional 
hours out of their very full schedules for this work? Table 7 presents the strategies employed by 
each SAU to provide time specifically allocated to LAS. 



Table 7. Strategies to Provide Specifically Allocated Time to an LAS 



How Time is Provided 


A of SAUs 


In-service days 


11 


Early release days 


9 


Release with subs 


9 


Teachers’ own time, compensated (summer, weekends, after school, etc.) 


9 


Teachers’ own time, uncompensated 


3 


Unknown 


2 


Note: N = 18; categories may overlap. 



It is clear that most SAUs are taking advantage of existing release-time mechanisms for 
LAS work. However, what is not so easily discernible is how much time is actually devoted to 
LAS. Characterizations of in-service days and department meetings often combine assessment 
with curriculum or other professional development topics. Additionally, committees in some 
SAUs meet four times a month while, in others, committees meet four times a year. In some 
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SAUs, the general faculty may only have one or two half days per year devoted to LAS, and the 
rest of their work is one-on-one with the curriculum coordinator or in groups with a teacher 
leader. One superintendent’s comment applies to most of the SAUs: “We decided we can’t come 
up with one game plan. We have to create a variety of ways for teachers to get the work done.” 
Resources 

SAUs have used a wide range of resources to accomplish LAS work. As one curriculum 
coordinator noted about the many resources provided by the MDOE: “If you take the time to 
participate in those opportunities, then you’ll be on track.” Table 8 presents the primary 
resources cited by the SAUs as valuable to accomplishing LAS-related work. 
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Table 8. Resources Cited as Valuable for Accomplishing LAS Work 



SAU 


MAP* 


LAD** 


LAS 

Guide 


MDOE 

Work- 

shops 


MDOE 

Person- 

nel 


Southern 

Maine 

Partner- 

ship 


Other 

SAUs 


Private 

Consul- 

tants 


Other 


1 


V 


V 


V 


V 




V 




V 




2 


V 


V 


V 




V 




V 


V 




3 


V 


V 




V 


V 






V 




4 


V 


V 


V 


V 








V 




5 


V 


V 


V 




V 


V 






V 


6 


V 


V 


V 








V 






7 


V 


V 






V 








V 


8 


V 


V 














V 


9 


V 


V 


V 


V 


V 


V 


V 






10 


V 


V 






V 










11 


V 


V 
















12 


















V 


13 






V 






V 








14 


V 


V 






V 


V 








15 


V 


V 














V 


16 


V 


V 






V 


V 






V 


17 




V 














V 


18 


V 


V 






V 








V 


*Maine Assessment Portfolio 
**Local Assessment Development 



As can be seen, MAP and LAD resources have been used by most of the participating 
SAUs. Usually, helpful Department personnel were identified by name. Private consultants were 
used primarily as resources for designing and implementing data management systems, although 
in at least one instance the consultant was advising on curriculum. Resources in the “Other” 
category were primarily websites, grant programs, or Measured Measures. 
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Compensation 

Sources of compensation generally cited by the SAUs are fairly unifonn: SAU budget, 
Title IIA, and Title V. However, compensation distribution varies considerably. In some SAUs, 
teacher leaders receive a flat annual sum while other teachers receive no compensation because 
they presumably will develop their materials in workshops, on early release days, and so on. In 
other SAUs, teachers are paid an hourly rate for LAS-related work on their own time or in after- 
school meetings. In almost all cases, SAUs recognize that the workloads of all teachers — not just 
teacher leaders — have increased in order to develop the local assessment system. 

Communication 

The frequency and manner of communication to the school board, staff, students, parents, 
and community varies by SAU. The data cannot be reduced to a single representation in tenns of 
frequency or formality; across SAUs, the approaches range from regularly published newsletters 
and monthly updates to sporadic presentations or incidental discussions. 

Communication with school boards usually occurs at board meetings. The frequency of 
such communication varies. For example, one SAU may provide monthly updates to the school 
board while another may only address the board when proposed policies are advanced. In both 
cases, the board is infonned as the process unfolds. Of the 16 SAUs responding to questions 
about communication, 10 report frequent communication with their boards, 5 report occasional 
communication, and 1 reports no communication. Usually, communications are verbal reports to 
the board at regularly scheduled meetings and are delivered by the superintendent, assistant 
superintendent, or curriculum coordinator. 
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All 16 SAUs report frequent communication to their staffs. This occurs through regularly 
scheduled professional development opportunities, meetings with curriculum coordinators or 
teacher leaders, email, and newsletters. 

Parents in 12 of the SAUs are reported to have received some level of LAS-related 
communication. Communication to parents may involve quarterly newsletters or a series of 
events (e.g., open houses, parent nights, regular newspaper articles). 

Communication to students, by either an administrator or teacher, occurs frequently in 
only three of the SAUs. Nine of the systems report occasional communication about LAS to 
students, and four report none. Some SAUs focus on incoming ninth graders: One LASIS 
researcher reported that letters “were sent to all ninth graders’ parents” and in another SAU that 
LAS “is discussed at Freshman Orientation.” As this time, many SAUs appear to have no formal 
communication policy for students. In the great majority of cases, most communication to 
students seems to be informal, ad hoc, and at the prerogative of individual teachers. 

Typically, an SAU further along in LAS development will have had more communication 
with its various constituents than an SAU still in the exploratory stages. The frequency, quality, 
and deliberateness of communication about LAS to stakeholders appears to be (from most to 
least): teachers, school board, parents, students. 

Summary 

Most SAUs have taken the pragmatic approach of using existing mechanisms, usually 
with adaptation, to develop their local assessment systems. Organizational structures with clearly 
defined responsibilities, and that include teachers, curriculum specialists, and administrators 
from all levels, appear to approach the work most efficiently. Such structures help the teachers 
by providing common time for collegial efforts, guide teachers to resources, facilitate 
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communication with various stakeholders, and coordinate the overall effort regarding LAS- 
related work. Equally important, solid organizational structure seems to promote a feeling of 
team spirit in the teachers; it is clear to them that they are part of a system-wide endeavor. 

Perceived “ Boosters ” and “Barriers” 

What are the perceived “boosters” regarding LAS work — i.e., those circumstances, 
resources, or events that are believed to facilitate the LAS development process? Similarly, what 
are the perceived “barriers” to this work — i.e., those circumstances, resources, or events that are 
believed to impede progress in LAS development? Perceived boosters or barriers seldom occur 
in isolation of each other. For example, where there is strong administrative support one might 
expect a generally positive attitude among staff (perhaps because adequate resources have been 
provided or because the leaders have provided encouragement and expressed recognition of work 
being accomplished). 

A reported booster or barrier necessarily reflects a perception of the salience of the 
booster or barrier relative to other facilitating or impeding forces. As will be seen, for example, 

14 SAUs indicated that time was a barrier. In fact, all 18 SAUs are grappling with time 
constraints, yet 4 SAUs apparently did not feel that this was as significant an impediment as 
other obstacles and, consequently, did not report time as a barrier. 

Boosters 

Table 9 presents the perceived boosters in descending order of frequency reported. 
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Table 9. Perceived “ Boosters ” Regarding LAS Work 



Boosters 


# of SAUs 
Reporting 
This Booster 


% of SAUs 
Reporting 
This Booster 


Dedicated teachers 


15 


83% 


MAP/LAD, LAS Guide, Measured Measures 


12 


67% 


Leadership participation in LAS development 


11 


61% 


Leadership continuity 


10 


55% 


Strong centralized coordination 


10 


55% 


Leadership (administration) support 


9 


50% 


Positive attitude 


9 


50% 


Inter- and intra-SAU collaboration 


9 


50% 


Internalization (not just responding to state/federal regulations) 


9 


50% 


MDOE personnel 


6 


33% 


Clearly articulated vision (by the leaders) 


4 


22% 


Note: N= 18; categories may overlap. 



Dedicated teachers. As Table 9 shows, dedicated teachers — professional educators who 
care passionately for their work, their charges, and the quality of instruction they provide — are 
the most frequently reported booster. Interestingly, three SAUs provided no indication through 
comments or actions that the dedication of their teachers is an important element in developing a 
local assessment system. It is possible that, in these SAUs, LAS work has not yet progressed to 
the stage of teacher involvement. 

MAP/LAD, LAS Guide, Measured Measures. Two thirds of the SAUs reported 
“MAP/LAD, LAS Guide, Measured Measures ” as a booster, making these resources the second 
most frequently perceived booster. One curriculum coordinator referred to the LAS Guide as her 
“Bible,” saying she consulted it all the time. Those who indicated that these resources were 
boosters typically used the material as examples or templates, particularly MAP and LAD tasks. 
However, some SAUs found less utility with MAP and LAD because they were expecting these 
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assessments to already align with their local curriculum. And in three SAUs, there was little or 
no evidence that personnel were even aware of these sites. 

Why didn’t all SAUs find these tools worthy of booster status? Regarding the LAS Guide, 
some felt that its publication date (June 2003) was late, given the LAS progress that SAUs were 
supposed to be making. Others felt that its content was confusing. However, it should be noted 
that the SAUs not reporting the LAS Guide as a booster seem to have discounted its utility and 
have not taken full advantage of it. Conversely, those SAUs that have studied the LAS Guide, 
distributed it widely, and incorporated it in their LAS process have found it to be immensely 
helpful, even if some portions of it were seen as confusing. 

Leadership participation in LAS development. Each SAU that reported “leadership 
participation” as a booster or was observed to have this characteristic has had continuous hands- 
on involvement by the administrative leadership. In these SAUs, the administrative function goes 
beyond setting policy, goals, and timelines. Administrators participate in the development of 
curriculum and the discussions about assessment. Their expertise and understanding of the 
ramifications of local assessment on policy, teachers, students, and the community informs the 
entire process. Even when LAS work reflects primarily a bottom-up approach, knowledgeable 
involvement by top leadership provides the staff with direction and validation of their work. 

Leadership continuity. SAUs with minimal leadership turnover seem to have the 
advantage of administrators who are known and trusted by faculty and the community, and 
whose administrators possess a familiarity with their systems that facilitates LAS work. Not 
unusually, such SAUs have already been involved in systemic change. They may have already 
developed structures or processes for aligning curriculum with Learning Results or for electronic 
data management. One curriculum coordinator reports that “because [the SAU personnel] have 
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been involved in this type of work for over five years, most groups of teachers and facilitators 
have developed their own structure for meeting times.” Often, they have fostered a culture of 
constant improvement and revision. The teachers seem to view LAS as a continuation of their 
efforts rather than as some new unrelated requirement. 

Strong centralized coordination. Typically, overall coordination of LAS development 
falls to either an assistant superintendent or the curriculum specialist who then delegates 
responsibilities to principals and/or teacher leaders. Strong coordination should not be confused 
with autocratic, top-down management, which can alienate teachers from the effort. Rather, 
strong coordination seeks to facilitate the LAS work by providing a fonnal structure for support 
and for reporting. Attributes of strong centralized coordination apparent in the SAUs in this 
study include clearly articulated responsibilities, reasonable but firm timelines, regularly 
scheduled meetings, and regular communication — both formal and infonnal. 

Leadership ( administration ) support. Interviewees from all participating SAUs agree on 
the importance of leadership support, but only half of the reporting SAUs indicate they had such 
support. There are four primary characteristics of this support in these SAUs: provision of time, 
availability of financial resources, provision of professional development opportunities, and 
promotion of positive attitudes. As seen above, strategies for making time available are fairly 
conventional. However, the frequency with which teachers are released during the school day or 
are encouraged (through compensation) to work outside of school hours may be one indicator of 
the level of support. Closely connected to this, of course, is the provision of financial resources. 

It takes money — as stipends, hourly compensation, or for substitutes — to make time available for 
teachers. Adequacy of funding is also a factor in professional development opportunities, 
particularly when this involves hiring consultants or release time to travel to workshops outside 



Local Assessment System Implementation Study ( 2003-2004 ): Technical Report 

August 2004 




24 



the SAU. However, there are professional development and training opportunities that don’t 
necessarily involve significant additional expenditures. Scheduling LAS training for inservice or 
early release days, disseminating infonnation and examples via e-mail, Web sites, and 
newsletters have been used. Finally, administration shaping of positive attitudes toward the work 
of LAS development appears to have value. Teachers identify with their SAU community and 
feel that they are contributing to a group effort valued by all. 

Positive attitude/internalization. While it may not hold that teachers with positive 
attitudes always produce better work than teachers with less positive views, one can reasonably 
assume that positive attitudes create a more conducive climate for getting that work done. 
Teachers who demonstrated a positive attitude towards LAS work typically felt that the work 
was valuable, that their administrators and colleagues valued it, and that it would improve their 
effectiveness as educators. If a positive attitude is a necessary precondition for internalization, 
one cannot “make the work his own” if the work is not seen as valuable. At a minimum, it seems 
that teachers’ positive attitudes toward LAS and internalization of the process and concept 
facilitate the development work. As one curriculum coordinator said, “Our goal is to make this 
work authentic. And that helps with the attitude tremendously. If teachers aren’t provided time to 
talk about what this means, then they don’t make this real work that can be used in the 
classroom.” 

Inter- and intra-SAU collaboration. Half of the participating SAUs cited collaboration, 
either within the SAU or with other SAUs, as a booster to their LAS work. Within a given school 
system, collaboration between teachers may have begun as part of a structured development 
process. However, once teachers “opened their classroom doors,” a deeper collegiality began to 
develop. The need to work together to craft common assessments has in many cases transfonned 
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into a desire to share teaching techniques or resources. Whether the need to collaborate on LAS 
encouraged greater collegiality or, rather, an existing culture of strong sharing facilitated LAS 
development, where such a culture exists it is viewed as a booster to the LAS effort. 

As reported above, LASIS sites convened in November 2003 to discuss and complete a 
survey, ask questions about LASIS, and to talk with other participating SAUs about their LAS 
work. It is not known whether the November meeting stimulated new collaborative activities 
beyond those at the meeting, but it is clear that where such relationships exist, the interacting 
SAUs view inter-SAU activities positively. In some instances, a given SAU may function more 
as a trainer or provide policies and blueprints as templates. In at least one SAU, teachers from 
five different systems meet by grade cluster or content area to build common assessments. 

MDOE personnel. While only six SAUs mentioned Maine Department of Education 
personnel as being boosters, nine referred to these same personnel as a resource (see Table 8). 
Often, MDOE personnel are mentioned by name as being particularly helpful or knowledgeable. 
As one curriculum coordinator replied when asked about Department resources: “Very helpful. 
DOE has provided work all along the way. [names person] does a tremendous job, so does 
[names person]. We have participated in all the work. Districts that don’t know what they’re 
supposed to do probably haven’t stayed the course. When you stick with the work, it doesn’t 
look so overwhelming.’’ It appears that SAUs whose LAS progress is further along the 
continuum tend to have had more interaction with MDOE personnel and, therefore, have greater 
appreciation for them. 

Clearly articulated vision (by the leaders ). As a rule, those SAUs reporting or evidencing 
a clearly articulated vision also exhibited a teaching staff with elan, positive attitudes about their 
work, and belief in the desirability of LAS. Typically, where a positive vision was reported, that 
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vision was articulated by the administration: “LAS is simply one more aspect of good teaching 
and has been promoted in that way.” 

Barriers 

As indicated above, “barriers” are defined as those circumstances, resources, or events 
believed to impede progress in LAS development. Table 10 presents these barriers, in 
descending order of the frequency reported. 



Table 10. Reported or Observed Barriers that Impeded the LAS Development Process 



Barriers 


N of SAUs 
Reporting 
This Barrier 


% of SAUs 
Reporting 
This Barrier 


Time 


14 


78% 


Scope of work 


6 


33% 


Lack of administrative support 


5 


28% 


Confusion about requirements/unclear definitions 


5 


28% 


Money 


4 


22% 


Lack of structure 


4 


22% 


Cynicism about LAS process and/or requirement 


4 


22% 


Administrative turnover 


4 


22% 


Resistance to change 


3 


17% 


Lack of DOE formal guidance and/or history of DOE rule changes 


3 


17% 


Note: N= 18; categories may overlap. 



Time. Many SAUs have attempted to fit the work of LAS into the school calendar by 
using the traditional mechanisms of early release days, in-service days, and time set aside for 
committee work. In almost all cases, those interviewed report that this has been insufficient. 
Teachers and administrators find themselves engaged in LAS development as an extracurricular 
activity — outside of school hours — and not always with compensation. In addition to solitary 
effort, teachers also need to find the time to meet with each other to agree on common 
assessments and undertake common scoring. 
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Coordinating schedules with other teachers from other schools in the SAU to conduct this 
work has proved a fonnidable task. Juggling hours to accommodate LAS-related work while still 
performing all of the other responsibilities of public educators has been challenging for the 
teachers. One participant reported that “Teachers are watery eyed and ready to break” and “feel 
like a tidal wave has rolled over them.” As another put it, teachers are at the “tail end of being 
enthusiastic.” Without a doubt, insufficient time is the single greatest barrier to LAS 
development. 

Scope of work. Scarcity of time is a function of the many other responsibilities educators 
have as well as the scope of LAS work itself. One administrator commented, “We have overdone 
taking teachers out of the classroom. If the state’s plan is to let teachers take time out of the 
classroom for scoring, we will have even greater problems statewide.” Yet, fewer than half of 
those who cited time as a barrier specifically mentioned the amount of work to be done as a 
barrier. Why? It may be that in some SAUs the work has been organized or “chunked” in such a 
way as to make it manageable. In other words, in those SAUs exhibiting a high degree of 
structural organization and delineation of responsibility, the task at hand seems less 
overwhelming than in those SAUs where the teachers feel that they are operating on their own, 
with minimum guidance, minimum support, and minimum peer interaction. One might postulate 
that it is not the amount of work that is the barrier but the perception that the work seems 
intenninable and insurmountable. Even as some teachers are developing common assessments 
they are anticipating the increased workloads that LAS will necessitate. As one teacher 
calculated: “If we do two assessments per student per core area, that’s 6,000 assessments. For 
twenty teachers, 300 tasks per teacher!” One administrator stated simply, “You’re asking too 
much.” 
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Lack of administrative support. By lack of administrative support, we mean the failure 
of administrators to provide staff with resources, processes, or rationales to accomplish the work. 
In at least one SAU, there seems to be almost no involvement by administrators at all and, in the 
void, a self-selected team of teachers emerged to provide impetus and focus to the LAS work. 
Here, leadership certainly is present due to the effort of these teachers, yet there seems to be little 
direction or support from the administrative level. Thus, this SAU exhibits a vacuum of 
leadership that is being filled by teachers, but it lacks administrative support nonetheless. 

Confusion about requirements/unclear definitions. Most SAUs indicated that confusion 
at the administrative level concerning LAS requirements has largely been eliminated because of 
clarification through consulting with MDOE personnel. In most cases where confusion was still 
indicated, it was teachers needing additional professional development about the requirements or 
definitions. What are bundles? What is balance of representation? What does “ Meets or exceeds 
the standard” mean? Clearly, these are issues arising from lack of communication and/or 
professional development. Many SAUs report initial confusion gradually giving way to 
comprehension as curriculum specialists work with MDOE personnel. In most cases, SAUs in 
earlier stages of LAS development and implementation were more likely to report confusion 
about requirements or definitions. From a high school teacher: “The state needs to figure out 
what it wants.” Usually such confusion seemed to dissipate once curriculum specialists had the 
opportunity to work with MDOE personnel and to share the results with their teachers. 

Money. Surprisingly, only four SAUs specifically identified the lack of money as a 
barrier to LAS development. SAUs have utilized budgeted money from their systems as well as 
various state and federal sources, including grants, to support LAS efforts. In one instance, 
however, one superintendent who did not feel that money was a constraint believed that LAS 
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work was part of the teachers' responsibilities as professional educators; consequently, he saw no 
need to provide compensation to teachers for work accomplished outside the regular school day. 
SAUs deploy their financial resources to hire consultants, pay stipends to teachers, pay for 
substitutes to provide teachers release time for LAS development tasks, and integrate data 
management systems. 

Lack of structure. How is the work getting done? Can responsibilities be represented 
graphically in a flow chart or organizational chart? Is communication regular and formalized? 
Are regular and specific times for meetings scheduled? The 18 sites exhibit structures that range 
from formally instituted to almost ad hoc, informal structures. Those systems whose efforts are 
not channeled through clearly defined lines of responsibility and lack strong central coordination 
seem to also have low LAS-related morale, poor communication to stakeholders, and absence of 
established LAS policies. In some instances this relationship may be illusory because the SAUs 
may not yet have arrived at the point where meaningful communication or policy formulation is 
possible. 

Cynicism about LAS. Where cynicism about LAS work surfaced, it typically involved 
one of two perceptions. The first perception is that a “local” assessment system is a hollow 
gesture towards local control — that all LASs ultimately will conform to a state-mandated form. 
As one individual asked, “Are we allowed to develop our own or not? If you’re going to impose 
it, impose it.” And another: “It’s called a local system, but it looks like the State’s system.” The 
second perception is that the LAS process is untenable and the product will be unusable. A 
teacher had this to say: “It won’t be real. It won’t be useful. It won’t be honest.” Behind such a 
belief appears to be the sentiment that authentic teaching does not need, nor can it improve from, 
such a system. 
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Cynicism about LAS was found in almost all participating SAUs to one degree or 
another. In most cases, however, such sentiments constitute little more than undercurrents having 
little impact on the work at hand. But in four SAUs the cynicism was more acute, which raises 
troubling questions regarding LAS development over the long term in these four sites. 

Administrative turnover. Four SAUs reported that administrative turnover has had a 
negative impact on their LAS efforts. Development of a local assessment system occurs within 
the context of other initiatives to strengthen teaching and learning. Superintendents and 
curriculum directors are always at the forefront of these efforts, yet any two administrators may 
approach the work in vastly different ways — philosophically, organizationally, emotionally. 

Lack of continuity in leadership may have caused changes in emphasis or technique or culture, 
which eroded the confidence and enthusiasm of staff. In at least one SAU, staff cited changed 
agendas as proof that their previous efforts were wasted. Worse, they may conclude that the new 
work they undertake also will not be worthwhile. In addition to this perceptual dimension, there 
is also the very real possibility that work previously expended was simply a false start. Teachers 
find themselves owning devalued products or processes and expected to rebuild according to the 
latest blueprint. In terms of constructing an LAS, those school systems having suffered 
administrative turnover may be poorly positioned for the effort because other systemic initiatives 
may be wanting (e.g., curriculum mapping or Learning Results alignment). More insidious, the 
teachers’ esprit and belief in their mission may have weakened due to the recurring crises, and 
they may no longer have faith in “The next big effort.” 

Resistance to change. Resistance to change, in most cases, really means unwillingness to 
accept the change as beneficial. For example, consider the principal who said, “There’s no 
human side to teaching anymore,” or another who reported of his teachers, “Many feel they are 
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testing more than teaching.” Interestingly, teachers talked in terms of desirability of the change. 
Not one teacher interviewed evinced resistance to change when there was a concomitant belief 
that the change would benefit students. True, teachers are apprehensive about ceding even more 
classroom autonomy to state or federal mandates, but they repeatedly indicated a willingness to 
do so when they were convinced such change would ultimately help students. In short, resistance 
to change is trumped by belief in the value of the change. But a minority of the teachers 
interviewed doesn’t view this particular change — LAS — as beneficial. One curriculum 
coordinator lamented, “We’re funneling people’s energies away from the matter at hand 
[teaching children].” In those instances where resistance was reported, one might do well to 
examine issues of leadership and school culture. As another curriculum coordinator noted, 

“There isn’t really a negative attitude towards LAS as there is simply an entrenchment of doing 
things the old way.” 

Lack of MDOE formal guidance and/or history of DOE rule changes. As previously 
noted, the MDOE efforts to support LAS work through workshops, the LAS Guide, training 
materials, and ongoing consultation are reported by many participating SAUs to be resources and 
boosters. Nevertheless, several SAUs were frustrated by what was viewed as a lack of DOE 
guidance (at least before the LAS Guide and Considering Consistency came out), or an 
inclination to append new requirements or change its rules. As one individual asked, “Why do 
we bother doing this if the rules are just going to change?” A group of high school teachers in 
one SAU felt that they were “ahead of the State [in LAS development] and then had to go back 
and change work because of new rules.” 
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Phone Interviews and Site Visits: Distillation of Key Findings 

1 . A successful organizational model has most or all of these characteristics: 

• Strong central coordination with clearly delineated responsibilities 

• Participation by administrators at both the SAU and building level, curriculum 
specialists, teachers, and teacher leaders 

• Committees that hold regularly scheduled meetings and report on these meetings 
to administrators and staff 

• Designation of time for teachers to specifically learn about and work on LAS is an 
organizational priority 

• Communication to teachers and stakeholders is more than informal and includes a 
variety of media 

2. Boosters for LAS development are: 

• Dedicated teachers whose professionalism impels them to contribute to the effort, 
whether or not they have accepted the premise of LAS 

• Systematic and deliberative use of MAP/LAD and the LAS Guide 

• Leadership participation in LAS development, by administrators and teacher 
leaders, resulting in an LAS that is cohesive and well-integrated with curriculum 

• Leadership continuity that folds LAS development into other systemic change 
efforts 

• Strong centralized coordination that clearly delineates responsibility, fosters 
communication, and monitors progress 

• Leadership (administration) support in provision of time, financial resources, 
professional development opportunities, and promotion of positive attitudes 
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• The internalized belief among administrators and teachers alike that the LAS 
work is meaningful and good for students — that it’s not merely a response to an 
external mandate 

• Inter- and intra-SAU collaboration that allows for the sharing of policies, sample 
assessments, common problem solving, and common scoring, as well as fostering 
increased collegiality among teachers 

• Drawing on the services and expertise of MDOE personnel 

• Clearly articulated vision (by the leaders) that renders the work necessary, 
important, and desirable 

3. Barriers to LAS development are: 

• Lack of time for professional development and assessment creation 

• Scope of work that seems too vast and too complicated to be accomplished, given 
existing responsibilities 

• Lack of administrative support, whether intentionally or not, in supplying 
personnel with necessary resources 

• Confusion about requirements/unclear definitions, resulting in SAU false starts or 
lack of confidence regarding LAS development 

• Lack of money for professional development, stipends, and personnel to support 
LAS work 

• Lack of organizational structure to coordinate LAS development 

• Cynicism about LAS 

• Administrative turnover that deprives SAUs of stability, continuity of vision and 
culture, and established mechanisms for effecting systemic change 
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• Resistance to change 

• Perceived lack of MDOE formal guidance and/or history of changing the rules 

At this point in LAS development, by far the most cited barrier is insufficient time for 

teachers to get professional development and/or to create (or select) the assessments. This barrier 
is reported in those SAUs that seem to have enjoyed the most success as well as in those SAUs 
that appear to be in the early stages of LAS development. The time barrier seems to be directly 
related to, and perhaps exacerbated by, other reported barriers to LAS development: scope of the 
work, lack of administrative support, lack of money, and lack of organizational structure for 
doing this work. 

The perception of barriers in any SAU may not be a good indicator of SAU progress 
toward LAS implementation. More pertinent, perhaps, are the perceptions of boosters that 
arguably attenuate these barriers. At least 82% (9 out of 1 1) of the reported or observed boosters 
occurred in 50% or more of the SAUs; no booster was reported in all of the SAUs. Thus, LAS 
progress does not appear to require any one booster or mix of boosters. And, of course, just as 
important as a booster’s presence is the quality of that booster and the manner of its employment. 
Nevertheless, the eight SAUs that seem furthest along in LAS implementation share these four 
boosters: 

• dedicated teachers 

• use of MAP/LAD and the LAS Guide 

• leadership participation in LAS development 

• strong centralized coordination 
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THE PEER REVIEW OF LAS DOCUMENTATION 

The 18 LASIS sites were invited to participate in a peer review session held in Waterville 
on May 17 - 18, 2004. While 16 of the 18 sites were represented at the session, only 14 sites 
brought LAS documentation for review. 

As indicated earlier in this report, the peer reviewers’ charge was to determine the degree 
to which there was evidence in the SAU’s LAS materials that progress had been made in 
enacting the various principles and criteria embodied in the LAS Guide. This component of 
LASIS had several purposes: (a) each site would benefit from the task of assembling its LAS 
materials — an “inventory taking” of sorts; (b) each site would benefit from the peer review 
itself — an appraisal of their LAS progress, by peers and based on submitted evidence; (c) 
participants would be become familiar with, and obtain, a helpful tool for self-assessment as they 
continue to their LAS work; and (d) the resulting quantitative ratings would be an additional 
source of data for the present study. It also was felt that participants simply would be interested 
to see how other SAUs were approaching the challenges of LAS development. 

Prior to the peer review session, participants were provided the peer review instrument as 
well as a checklist of materials that would be needed at the meeting. (The peer review 
instrument and checklist are presented in Appendices D and E, respectively.) Once at the 
meeting, each participant was assigned to a table with five or six participants from other sites to 
form mixed-site review groups. Each group also included a table leader. The table leaders, who 
were members of either the MDOE or the Maine Mathematics and Science Alliance, recorded 
participants’ feedback and provided direction and/or clarification when needed. 
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Each site’s LAS materials were reviewed 2-3 times, each time by a different mixed-site 
group of individuals using the peer review instrument. 7 The reviewers’ charge was to determine 
the degree to which there was evidence in the SAU’s LAS materials that progress had been made 
in enacting 1 1 principles and criteria embodied in the LAS Guide. The reviewers rated an SAU’s 
evidence on a scale of 1 to 5, where “1” signified that the evidence did not demonstrate 
accomplishment of any aspect of the principle/criterion; “3” indicated that some, but not all, 
aspects of the principle/criterion had been demonstrated in the evidence; and “5” signified that 
the evidence clearly demonstrates accomplishment of all aspects of the principle/criterion. 
(Values of “2” and “4” were left without descriptions.) 

Subsequent interrater-agreement analyses raised questions about the defensibility of the 
three middle values on the 5-point scale. Consequently, for analysis purposes this scale was 
reduced to a 3-point scale, with 2s and 4s subsumed by the middle point. On the reduced scale, 
then, “1” signified that the evidence did not demonstrate accomplishment of any aspect of the 
principle/criterion; “2” (which combines the questionable three middle values of the original 5- 
point scale) indicated that some, but not all, aspects of the principle/criterion had been 
demonstrated in the evidence; and “3” signified that the evidence clearly demonstrates 
accomplishment of all aspects of the principle/criterion. 

As an estimate of reliability, the percentage of agreement (on the reduced 3 -point scale) 
was calculated across the 1 1 principles/criteria for all 14 sites. First, the percentage of agreement 
was calculated between ratings obtained during Session A and those obtained during Session B 
for each principle/criterion. This process was then duplicated for Session A and Session C 



7 Below, the separate review sessions for an SAU’s documents are referred to as Session A (first review), Session B 
(second review, by a different group), and Session C (third review [if applicable], by a different group still). 
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ratings and for Session B and Session C ratings. All percentages of agreement were then 
averaged to obtain the overall percentage of agreement, which was 69%. 

It appears that agreement was influenced by the manner in which LAS materials were 
organized by each SAU. Where an SAU’s materials were not organized well, for example, some 
reviewers undoubtedly made charitable assumptions about the presence of documented evidence 
(e.g., “I’m sure it’s in there somewhere. . .”) whereas other reviewers did not. If true, then the 
peer review process could be enhanced in the future by providing more specific guidelines to 
SAUs for organizing their LAS materials for review, as well as providing more emphatic 
instructions to reviewers for evaluating the documentation of evidence regarding LAS 
development 

Table 1 1 presents mean ratings for the 1 1 principles/criteria across all sites, along with 
the percentage of agreement for each principle/criterion. With means ranging from 1.54 to 2.17, 
one sees that for no principle/criterion is the mean as low as the minimum “1.00” or as high as 
the maximum “3.00.” Not surprisingly, the highest mean ratings correspond to principles/criteria 
regarding early aspects of LAS development, such as selecting/constructing common 
assessments and their scoring guides. Similarly, the lowest mean ratings correspond to 
principles/criteria regarding later aspects of LAS development, such as the formulation of policy 
concerning replacement, performance standards, and reporting. An exception is the low mean 
rating for Comparability: Policy documents for alternate assessments/accommodations and 
sample assessments, which arguably should not be viewed as a “later” aspect of LAS 
development. In any case, these means should be treated cautiously, particularly where interrater 
agreement is relatively low. 
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Table 11. LAS Peer Review ( 14 sites): Mean Rating and Interrater Agreement for each 
Principle/Criterion 





Mean 


Standard 

Deviation 


Interrater 

Agreeement 


Coherence and Fairness: Curriculum documents, course of study documents, 
communication guidance on pathways to ensure opportunity to leam 


2.00 


.36 


80% 


Coherence and Fairness: Five sample assessments (including locally developed or adapted 
assessments) from across 5 content areas and 3 grade spans; curriculum and instruction 
artifacts used to develop skills and knowledge for each assessment 


2.01 


.42 


77% 


Sufficiency through Assessment Type Selection and Distribution: Description of process, list 
of prioritized performance indicators 


1.87 


.45 


54% 


Sufficiency through Assessment Type Selection and Distribution: Content area/grade span 
templates 


1.99 


.23 


90% 


Comparability: Templates with common assessments labeled for the five content areas and 
three grade spans 


2.17 


.65 


62% 


Comparability: Policy documents for alternate assessments/accomodations and sample 
assessments, two sample IEPs 


1.54 


.57 


73% 


Comparability: Sample assessments with scoring guides if used, conversion protocol 


2.08 


.50 


67% 


Replacement: Replacement policy 


1.54 


.77 


83% 


Performance Standards: Performance standards set/policy document 


1.71 


.71 


57% 


Public Reporting: Policy for reporting, sample reporting format 


1.67 


.55 


49% 


Chapter 127 Sec. 4.02: Graduation policy, policy regarding MEA, professional development 
plan, communication strategy plan, data management plan 


1.80 


.37 


63% 



In addition to completing the peer review forms, participants also were invited to 
complete a feedback form regarding the peer review instrument and the experience of peer 
review itself The feedback form is presented in Appendix F. The intent of the feedback forms 
was to assist the MDOE in revising, if warranted, the peer review process and instrument before 
sharing it with other Maine SAUs. 

The feedback form comprised nine selected-response items and two opened-ended items. 
Reviewers were also invited to provide written comments for any selected-response item. Table 
12 summarizes the quantitative data from this form. As these data show, the reviewers generally 
found the instrument helpful and the process informative. Indeed, at least three quarters of the 
peer reviewers responded in the affirmative to all questions. 
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Table 12. Peer review feedback form: quantitative results. 







Yes 


Somewhat 




No 




n 


% 


n 


% 


n 


% 


Were the AM training and materials helpful with your use of the 
review tool? 


37 


75.5% 


ii 


22.4% 


i 


2.1% 


Did it make sense to have the peer review tool organized by 
principles and criteria as is done in the LAS Guide? 


45 


90.0% 


5 


10.0% 


0 


.0% 


Were the guiding Peer Reviewer Questions clear? 


38 


77.5% 


10 


20.4% 


1 


2.1% 


Did the evidence column clarify what you needed to document 
in order to complete your rating? 


41 


82.0% 


9 


18.0% 


0 


.0% 


Was the use of the 1-5 rating scale helpful for the task at hand? 


40 


81.6% 


9 


18.4% 


0 


.0% 


Was the supporting and missing evidence appropriate feedback? 


44 


86.3% 


7 


13.7% 


0 


.0% 


Was the process of scoring individually and then coming to 
consensus effective and efficient? 


35 


74.5% 


11 


23.4% 


1 


2.1% 


Did the process allow you to identify accomplishments and next 
steps for the SAUs you reviewed? 


44 


86.3% 


6 


11.8% 


1 


2.0% 


Will the information you have received for you SAU help you 
move forward? 


42 


81.3% 


4 


8.7% 


0 


.0% 



On 7 of the above items, 1 -5 participants found the item "not applicable." 



The supplementary comments offered by the reviewers also reflected a positive appraisal 
of the peer review instrument and process. For example, in response to Will the information you 
have received for your SAU help you move forward?, one respondent answered, “Definitely — I 
will return and meet with my assessment team as to what we need to do, and the results of our 
peer review will provide this information.” As another example, when asked What else could a 
process like this provide?, one reviewer stated, “Nothing — It was great!” 

While the majority of responses were positive, the concern most often expressed by 
reviewers focused on the organization of materials. A few reviewers indicated that they would 
prefer to have had the peer review tool provided to them more in advance to facilitate assembling 
the materials. And one reviewer stated, “We needed to have the AM session ‘2 weeks ago’ so 
that there would have been more consistency in the organization of our materials.” Another 
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reviewer replied, “It would be helpful to have an exemplar or prototype available to help 
organize materials in a standard fashion so that the review process can be streamlined.” 

The comment of one reviewer may have summed up the peer review meeting most 
succinctly — “Clarifying and exhausting.” 

THE JUNE 2004 SURVEY 

LASIS participants received a comprehensive survey in June 2004 that assessed their 
progress in, and beliefs about, LAS development (see Appendix G). The results of this survey 
provide the most current portrait of implementation status in LASIS sites. Twelve of the 18 

o 

LASIS sites responded to the survey, with one SAU completing two surveys as instructed. 

Respondents were invited to provide information regarding who is involved, and to what 
extent they were involved, in LAS development and implementation. Information was also 
solicited regarding the status of particular LAS-related tasks in the five content areas 
(English/language arts, mathematics, science and technology, social studies, and health and 
physical education) and at each of three grade spans (K-4, 5-8, and 9-12). A few items on the 
survey invited respondents to provide a more comprehensive view of the implementation status 
by reporting implementation status either across particular tasks or across grade spans. 

In addition to implementation status, data were collected regarding collaboration among 
SAUs and the level of help needed to complete certain tasks. Respondents’ perceptions of the 
understanding of graduation requirements by ninth-grade students and their parents were also 
collected. As with previous LASIS data collection activities, respondents were invited to identify 



s Because some SAUs are comprised of Unions which could have separate Local Assessment Systems for each of 
the schools, SAUs were instructed to complete a survey for each discrete LAS in place within their SAU. 
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boosters and barriers to the development/implementation of the LAS, with the additional stipulation 
that “time” and “money” not be listed (since this was well established by now). 

Because of the small number of respondents (n = 13), care must be taken when interpreting 
the percentages in the tables that follow. Variation in even a single response renders a large 
differential in the percentages displayed. 

Using a four-point scale, ranging from no involvement (1) to extensive involvement (4), 
respondents rated the extent to which specific roles or structures were involved with the 
development and implementation of their LAS. (If a particular role or structure did not exist in the 
SAU, respondents indicated so.) Table 13 shows that over half of the respondents indicated their 
SAU had a curriculum coordinator, a K-12 content area team, and/or a local assessment team who is 
extensively involved in LAS development and implementation. The LAS-involvement of the 
superintendent varies considerably across LASIS sites, as does the principal (although to a lesser 
extent). All but one of the respondents indicated their SAU did not have an assessment specialist. 
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Table 13. Extent of Involvement in the Development and Implementation of the LAS 





This role or 
structure does 
not exist 


No 

involvement 

1 




2 




3 


Extensive 

involvement 

4 


n 


% 


n 


% 


n 


% 


n 


% 


n 


% 


Assessment specialist 


12 


92.3% 


0 


.0% 


0 


.0% 


0 


.0% 


i 


7.7% 


Assistant superintendent 


9 


69.2% 


1 


7.7% 


0 


.0% 


0 


.0% 


3 


23.1% 


Communications personnel 


10 


83.3% 


2 


16.7% 


0 


.0% 


0 


.0% 


0 


.0% 


Content leader 


6 


46.2% 


0 


.0% 


1 


7.7% 


1 


7.7% 


5 


38.5% 


Curriculum coordinator 


6 


46.2% 


0 


.0% 


0 


.0% 


0 


.0% 


7 


53.8% 


Data management specialist 


8 


66.7% 


1 


8.3% 


1 


8.3% 


1 


8.3% 


1 


8.3% 


Principal 


0 


.0% 


0 


.0% 


4 


30.8% 


5 


38% 


4 


30.8% 


Superintendent 


0 


.0% 


3 


23.1% 


3 


23.1% 


4 


31% 


3 


23.1% 


Other role 


0 


.0% 


0 


.0% 


1 


25.0% 


1 


25% 


2 


50.0% 


Curriculum committee 


2 


18.2% 


1 


9.1% 


1 


9.1% 


3 


27% 


4 


36.4% 


K-12 content area team 


0 


.0% 


0 


.0% 


2 


18.2% 


3 


27% 


6 


54.5% 


Local assessment team 


3 


25.0% 


0 


.0% 


1 


8.3% 


1 


8.3% 


7 


58.3% 


Other structure 


0 


.0% 


0 


.0% 


0 


.0% 


0 


.0% 


3 


100% 



Given a list of activities, respondents rated the progress being made in each activity for 
five specified content areas and three separate grade spans. A scale of 1 to 4 was employed, 
where “1” signified that no action had been taken on this activity yet, “2” indicated planning was 
in progress, “3” signified that the activity was partially complete, and “4” indicated that the work 
was complete. Tables 14a through 18c present this information, with each table comprising a 
single content area and grade span. 

These data suggest that the progress being made for these activities varies little from one 
content area to another. Insofar as English/language arts and mathematics originally had an 
earlier deadline for implementation than the other content areas, one may have expected 
activities in those two content areas to be closer to completion. Similarly, each of the three grade 
spans appears to be progressing at a similar pace. 
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Variation in progress surfaces when looking across the activities. Respondents were 
more likely to indicate that the tasks of developing curriculum documents aligned with Mcdne 
Learning Results and selecting a sampling of the performance indicators to assess using Mcdne’ s 
Bcdcmce of Representation or some other method of prioritization were completed than they were 
for the task of aligning instruction with curriculum and Maine Learning Results. The reason for 
this may be due in part to who is responsible for developing and implementing the LAS — i.e., 
curriculum coordinators and/or local assessment teams may have developed the curriculum and 
selected a sampling of performance indicators, but teachers have not actually aligned their 
instruction to this new curriculum. 

With few exceptions, approximately one third of respondents indicated that the task of 
being prepared to fully implement the expectations of the LAS in the fall of 2004 had been 
completed. Only 15.4% of respondents indicated that task had been completed for the 5-8 grade 
span in social studies. Conversely, 41.7% of respondents indicated that this same task had been 
completed in health and physical education at both the K-4 and 5-8 grade spans. 
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Table 14a. SAU’s Progress in English/Language Arts Grades K-4 





Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine Learning 
Results. 


2 


15.4% 


3 


23.1% 


8 


61.5% 


aligned instruction with curriculum and Maine Learning 
Results. 


2 


16.7% 


9 


75.0% 


1 


8.3% 


selected a sampling of the performance indicators to assess 
using Maine's Balance of Representation or some other 
method of prioritization. 


0 


.0% 


6 


46.2% 


7 


53.8% 


identified at least 8 assessments for each grade span that: 
samples every content standard; measures each content 
cluster five times; provides a variety of assessment types by 
clusters; identifies 75% of assessments as common. 


0 


.0% 


6 


46.2% 


7 


53.8% 


provided professional development in common scoring for 
teachers. 


2 


15.4% 


5 


38.5% 


6 


46.2% 


prepared to fully implement the expectations of the LAS in 
the fall of 2004. 


1 


7.7% 


8 


61.5% 


4 


30.8% 



Table 14b. SAU’s Progress in English/Language Arts Grades 5-8 





Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine Learning 
Results 


3 


23.1% 


2 


15.4% 


8 


61.5% 


aligned instruction with curriculum and Maine Learning 
Results 


2 


16.7% 


9 


75.0% 


1 


8.3% 


selected a sampling of the performance indicators to assess 
using Maine's Balance of Representation or some other 
method of prioritization 


0 


.0% 


6 


46.2% 


7 


53.8% 


identified at least 8 assessments for each grade span that: 
samples every content standard; measures each content cluster 
five times; provides a variety of assessment types by clusters; 
identifies 75% of assessments as common 


0 


.0% 


6 


46.2% 


7 


53.8% 


provided professional development in common scoring for 
teachers 


2 


15.4% 


5 


38.5% 


6 


46.2% 


prepared to fully implement the expectations of the LAS in the 
fall of 2004 


1 


7.7% 


8 


61.5% 


4 


30.8% 
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Table 14c. SAU’s Progress in English/Language Arts Grades 9-12 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned 
with Maine Learning Results 


i 


8.3% 


2 


16.7% 


2 


16.7% 


7 


58.3% 


aligned instruction with curriculum and 
Maine Learning Results 


0 


.0% 


2 


18.2% 


7 


63.6% 


2 


18.2% 


selected a sampling of the performance 
indicators to assess using Maine's Balance of 
Representation or some other method of 
prioritization 


0 


.0% 


0 


.0% 


5 


41.7% 


7 


58.3% 


identified at least 8 assessments for each 
grade span that: samples every content 
standard; measures each content cluster five 
times; provides a variety of assessment types 
by clusters; identifies 75% of assessments as 


0 


.0% 


0 


.0% 


5 


41.7% 


7 


58.3% 


common 


















provided professional development in 
common scoring for teachers 


0 


.0% 


3 


25.0% 


3 


25.0% 


6 


50.0% 


prepared to fully implement the expectations 
of the LAS in the fall of 2004 


0 


.0% 


1 


8.3% 


7 


58.3% 


4 


33.3% 



Table 15a. SAU’s Progress in Mathematics Grades K-4 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with 
Maine Learning Results 


0 


.0% 


2 


15.4% 


4 


30.8% 


7 


53.8% 


aligned instruction with curriculum and Maine 
Learning Results 


0 


.0% 


3 


23.1% 


8 


61.5% 


2 


15.4% 


selected a sampling of the perfonnance 
indicators to assess using Maine's Balance of 
Representation or some other method of 
prioritization 


0 


.0% 


1 


8.3% 


5 


41.7% 


6 


50.0% 


identified at least 8 assessments for each 
grade span that: samples every content 
standard; measures each content cluster five 
times; provides a variety of assessment types 
by clusters; identifies 75% of assessments as 


0 


.0% 


0 


.0% 


6 


46.2% 


7 


53.8% 


common 


















provided professional development in 
common scoring for teachers 


1 


7.7% 


2 


15.4% 


8 


61.5% 


2 


15.4% 


prepared to fully implement the expectations 
of the LAS in the fall of 2004 


0 


.0% 


1 


7.7% 


8 


61.5% 


4 


30.8% 
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Table 15b. SAU’s Progress in Mathematics Grades 5-8 





Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine Learning Results 


3 


25.0% 


3 


25.0% 


6 


50.0% 


aligned instruction with curriculum and Maine Learning Results 


3 


25.0% 


7 


58.3% 


2 


16.7% 


selected a sampling of the performance indicators to assess using 
Maine's Balance of Representation or some other method of 
prioritization 


1 


9.1% 


4 


36.4% 


6 


54.5% 


identified at least 8 assessments for each grade span that: samples 
every content standard; measures each content cluster five times; 
provides a variety of assessment types by clusters; identifies 75% of 
assessments as common 


0 


.0% 


5 


41.7% 


7 


58.3% 


provided professional development in common scoring for teachers 


4 


33.3% 


6 


50.0% 


2 


16.7% 


prepared to fully implement the expectations of the LAS in the fall of 
2004 


1 


8.3% 


7 


58.3% 


4 


33.3% 



Table 15c. SAU’s Progress in Mathematics Grades 9-12 





Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine Learning Results 


i 


8.3% 


3 


25.0% 


8 


66.7% 


aligned instruction with curriculum and Maine Learning Results 


i 


8.3% 


7 


58.3% 


4 


33.3% 


selected a sampling of the performance indicators to assess using Maine's 
Balance of Representation or some other method of prioritization 


0 


.0% 


5 


45.5% 


6 


54.5% 


identified at least 8 assessments for each grade span that: samples every 
content standard; measures each content cluster five times; provides a 
variety of assessment types by clusters; identifies 75% of assessments as 


0 


.0% 


5 


41.7% 


7 


58.3% 


common 














provided professional development in common scoring for teachers 


4 


33.3% 


5 


41.7% 


3 


25.0% 


prepared to fully implement the expectations of the LAS in the fall of 2004 


1 


8.3% 


7 


58.3% 


4 


33.3% 
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Table 16a. SAU’s Progress in Science and Technology Grades K-4 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine 
Learning Results 


0 


.0% 


4 


30.8% 


3 


23.1% 


6 


46.2% 


aligned instruction with curriculum and Maine Learning 
Results 


0 


.0% 


3 


23.1% 


8 


61.5% 


2 


15.4% 


selected a sampling of the performance indicators to 
assess using Maine's Balance of Representation or some 
other method of prioritization 


0 


.0% 


2 


15.4% 


4 


30.8% 


7 


53.8% 


identified at least 8 assessments for each grade span that: 
samples every content standard; measures each content 
cluster five times; provides a variety of assessment types 
by clusters; identifies 75% of assessments as common 


1 


7.7% 


1 


7.7% 


6 


46.2% 


5 


38.5% 


provided professional development in common scoring 
for teachers 


2 


16.7% 


3 


25.0% 


5 


41.7% 


2 


16.7% 


prepared to fully implement the expectations of the LAS 
in the fall of 2004 


2 


16.7% 


0 


.0% 


6 


50.0% 


4 


33.3% 



Table 16b. SAU’s Progress in Science and Technology Grades 5-8 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine 
Learning Results 


0 


.0% 


4 


30.8% 


3 


23.1% 


6 


46.2% 


aligned instruction with curriculum and Maine Learning 
Results 


0 


.0% 


3 


25.0% 


8 


66.7% 


1 


8.3% 


selected a sampling of the performance indicators to 
assess using Maine's Balance of Representation or some 
other method of prioritization 


0 


.0% 


1 


8.3% 


4 


33.3% 


7 


58.3% 


identified at least 8 assessments for each grade span that: 
samples every content standard; measures each content 
cluster five times; provides a variety of assessment types 
by clusters; identifies 75% of assessments as common 


1 


8.3% 


0 


.0% 


5 


41.7% 


6 


50.0% 


provided professional development in common scoring 
for teachers 


2 


18.2% 


3 


27.3% 


4 


36.4% 


2 


18.2% 


prepared to fully implement the expectations of the LAS 
in the fall of 2004 


1 


9.1% 


0 


.0% 


6 


54.5% 


4 


36.4% 
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Table 16c. SAU’s Progress in Science and Technology Grades 9-12 





Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine Learning Results 


2 


16.7% 


3 


25.0% 


7 


58.3% 


aligned instruction with curriculum and Maine Learning Results 


1 


8.3% 


7 


58.3% 


4 


33.3% 


selected a sampling of the performance indicators to assess using 
Maine's Balance of Representation or some other method of 
prioritization 


0 


.0% 


5 


41.7% 


7 


58.3% 


identified at least 8 assessments for each grade span that: samples 
every content standard; measures each content cluster five times; 
provides a variety of assessment types by clusters; identifies 75% of 
assessments as common 


0 


.0% 


6 


50.0% 


6 


50.0% 


provided professional development in common scoring for teachers 


5 


45.5% 


3 


27.3% 


3 


27.3% 


prepared to fully implement the expectations of the LAS in the fall of 
2004 


0 


.0% 


7 


63.6% 


4 


36.4% 



Table 17a. SAU’s Progress in Social Studies Grades K-4 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine 
Learning Results 


0 


.0% 


3 


23.1% 


3 


23.1% 


7 


53.8% 


aligned instruction with curriculum and Maine Learning 
Results 


0 


.0% 


2 


15.4% 


9 


69.2% 


2 


15.4% 


selected a sampling of the performance indicators to 
assess using Maine's Balance of Representation or some 
other method of prioritization 


0 


.0% 


1 


7.7% 


5 


38.5% 


7 


53.8% 


identified at least 8 assessments for each grade span that: 
samples every content standard; measures each content 
cluster five times; provides a variety of assessment types 
by clusters; identifies 75% of assessments as common 


0 


.0% 


3 


23.1% 


5 


38.5% 


5 


38.5% 


provided professional development in common scoring 
for teachers 


1 


7.7% 


5 


38.5% 


5 


38.5% 


2 


15.4% 


prepared to fully implement the expectations of the LAS 
in the fall of 2004 


0 


.0% 


2 


15.4% 


7 


53.8% 


4 


30.8% 
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Table 17b. SAU’s Progress in Social Studies Grades 5-8 





No action taken 


Planning in 


Partially 




Work 




on this yet 


progress 


complete 


complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine 
Learning Results 


0 


.0% 


3 


23.1% 


3 


23.1% 


7 


53.8% 


aligned instruction with curriculum and Maine Learning 
Results 


0 


.0% 


2 


15.4% 


9 


69.2% 


2 


15.4% 


selected a sampling of the performance indicators to 
assess using Maine's Balance of Representation or some 
other method of prioritization 


0 


.0% 


1 


7.7% 


6 


46.2% 


6 


46.2% 


identified at least 8 assessments for each grade span 
that: samples every content standard; measures each 
content cluster five times; provides a variety of 
assessment types by clusters; identifies 75% of 
assessments as common 


0 


.0% 


2 


15.4% 


7 


53.8% 


4 


30.8% 


provided professional development in common scoring 
for teachers 


1 


7.7% 


5 


38.5% 


4 


30.8% 


3 


23.1% 


prepared to fully implement the expectations of the LAS 
in the fall of 2004 


0 


.0% 


2 


15.4% 


9 


69.2% 


2 


15.4% 



Table 17c. SAU’s Progress in Social Studies Grades 9-12 



Planning in Partially Work 

progress complete complete 



Our SAU has... 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine Learning Results 


2 


16.7% 


3 


25.0% 


7 


58.3% 


aligned instruction with curriculum and Maine Learning Results 


1 


8.3% 


6 


50.0% 


5 


41.7% 


selected a sampling of the performance indicators to assess using 
Maine's Balance of Representation or some other method of 
prioritization 


0 


.0% 


5 


41.7% 


7 


58.3% 


identified at least 8 assessments for each grade span that: samples 
every content standard; measures each content cluster five times; 


0 


.0% 


7 


58.3% 




41.7% 


provides a variety of assessment types by clusters; identifies 75% of 
assessments as common 














provided professional development in common scoring for teachers 


4 


33.3% 


5 


41.7% 


3 


25.0% 


prepared to fully implement the expectations of the LAS in the fall of 
2004 


0 


.0% 


8 


66.7% 


4 


33.3% 
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Table 18a. SAU’s Progress in Physical Education Grades K-4 





No action taken 


Planning in 


Partially 


Work 




on this yet 


progress 


complete 


complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine 
Learning Results 


0 


.0% 


4 


30.8% 


4 


30.8% 


5 


38.5% 


aligned instruction with curriculum and Maine Learning 
Results 


0 


.0% 


4 


30.8% 


7 


53.8% 


2 


15.4% 


selected a sampling of the performance indicators to 
assess using Maine's Balance of Representation or some 
other method of prioritization 


1 


7.7% 


2 


15.4% 


3 


23.1% 


7 


53.8% 


identified at least 8 assessments for each grade span 
that: samples every content standard; measures each 
content cluster five times; provides a variety of 
assessment types by clusters; identifies 75% of 
assessments as common 


0 


.0% 


4 


30.8% 


4 


30.8% 


5 


38.5% 


provided professional development in common scoring 
for teachers 


1 


7.7% 


4 


30.8% 


6 


46.2% 


2 


15.4% 


prepared to fully implement the expectations of the LAS 
in the fall of 2004 


2 


16.7% 


2 


16.7% 


3 


25.0% 


5 


41.7% 



Table 18b. SAU’s Progress in Physical Education Grades 5-8 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine 
Learning Results 


0 


.0% 


3 


23.1% 


6 


46.2% 


4 


30.8% 


aligned instruction with curriculum and Maine Learning 
Results 


0 


.0% 


4 


30.8% 


7 


53.8% 


2 


15.4% 


selected a sampling of the performance indicators to 
assess using Maine's Balance of Representation or some 
other method of prioritization 


1 


7.7% 


1 


7.7% 


4 


30.8% 


7 


53.8% 


identified at least 8 assessments for each grade span 
that: samples every content standard; measures each 
content cluster five times; provides a variety of 
assessment types by clusters; identifies 75% of 
assessments as common 


0 


.0% 


2 


15.4% 


6 


46.2% 


5 


38.5% 


provided professional development in common scoring 
for teachers 


0 


.0% 


4 


30.8% 


7 


53.8% 


2 


15.4% 


prepared to fully implement the expectations of the LAS 
in the fall of 2004 


1 


8.3% 


2 


16.7% 


4 


33.3% 


5 


41.7% 



Local Assessment System Implementation Study ( 2003-2004 ): Technical Report 

August 2004 










51 



Table 18c. SAU’s Progress in Physical Education Grades 9-12 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed curriculum documents aligned with Maine 
Learning Results 


i 


8.3% 


i 


8.3% 


5 


41.7% 


5 


41.7% 


aligned instruction with curriculum and Maine Learning 
Results 


i 


8.3% 


2 


16.7% 


6 


50.0% 


3 


25.0% 


selected a sampling of the perfonnance indicators to 
assess using Maine's Balance of Representation or some 
other method of prioritization 


0 


.0% 


1 


8.3% 


4 


33.3% 


7 


58.3% 


identified at least 8 assessments for each grade span 
that: samples every content standard; measures each 
content cluster five times; provides a variety of 
assessment types by clusters; identifies 75% of 
assessments as common 


0 


.0% 


1 


8.3% 


7 


58.3% 


4 


33.3% 


provided professional development in common scoring 
for teachers 


0 


.0% 


3 


25.0% 


6 


50.0% 


3 


25.0% 


prepared to fully implement the expectations of the LAS 
in the fall of 2004 


0 


.0% 


2 


18.2% 


5 


45.5% 


4 


36.4% 



Respondents also were asked to rate the progress their SAU had made in Modern and 
Classical Languages, Visual and Performing Arts, and Career Preparation. Being mindful that 
these three content areas have a different timeline for implementation than the other content 
areas, respondents were asked only to provide a general rating of LAS progress. Specifically, 
respondents were instructed to reflect upon the various activities listed in Tables 14a through 18c 
and then provide a single rating at the three grade spans. 

As one might expect, respondents indicated that less progress had been made in these 
three content areas than in the five content areas described above. Further, there is little variation 
in LAS progress among grade spans in each area. More progress has been made in the area of 
Visual and Perfonning Arts than the other two areas. Tables 19a, 19b, and 19c present these 
findings in their entirety. 
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Table 19a. SAU’s Overall Progress in Developing and Implementing an LAS 
in Grades K-4 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Progress being made in... 


n 


% 


n % 


n % 


Modem and Classical Language 


7 


53.8% 


1 7.7% 


5 38.5% 


Visual and Performing Arts 


1 


7.7% 


3 23.1% 


9 69.2% 


Career Preparation 


4 


30.8% 


7 53.8% 


2 15.4% 



Table 19b. SAU’s Overall Progress in Developing and Implementing an LAS 
in Grades 5-8 



Progress being made in... 


No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


n 


% 


n 


% 


n % 


Modern and Classical Language 


5 


38.5% 


2 


15.4% 


6 46.2% 


Visual and Performing Arts 


1 


7.7% 


3 


23.1% 


9 69.2% 


Career Preparation 


4 


30.8% 


7 


53.8% 


2 15.4% 



Table 19c. SAU’s Overall Progress in Developing and Implementing an LAS 
in Grades 9-12 



Progress being made in... 


No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


n 


% 


n 


% 


n 


% 


n % 


Modern and Classical Language 


1 


8.3% 


3 


25.0% 


7 


58.3% 


1 8.3% 


Visual and Performing Arts 


1 


8.3% 


2 


16.7% 


9 


75.0% 


0 .0% 


Career Preparation 


3 


25.0% 


6 


50.0% 


3 


25.0% 


0 .0% 



Respondents also were asked to provide a general rating of their LAS progress across the 
K-12 grade span. Almost half of the respondents (46.2%) indicated they had completed the task 
of selecting either the recommended or alternative set of perfonnance standards from the LAS 
guide. As can be seen in Table 20, less progress had been made in the areas of public reporting, 
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developing policy for replacement assessments, and developing a comprehensive scoring system 
that will meet technical standards. 



Table 20. SAU’s Progress in Selected Activities in Grades K-12 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


Our SAU has... 


n 


% 


n 


% 


n 


% 


n 


% 


developed a policy for replacement assessments 


3 


23.1% 


5 


38.5% 


4 


30.8% 


i 


7.7% 


selected either the recommended or alternative set 
of perfonnance standards from the LAS guide 


0 


.0% 


5 


38.5% 


2 


15.4% 


6 


46.2% 


developed a procedure for public reporting of 
scores at the required levels for certification 


3 


23.1% 


5 


38.5% 


5 


38.5% 


0 


.0% 


developed a comprehensive scoring system that 
will meet technical standards 


0 


.0% 


2 


15.4% 


9 


69.2% 


2 


15.4% 



Respondents were asked to what extent their SAU was prepared to assess the progress of 
students with disabilities/ESL/504 toward achieving the Learning Results. As Table 21 shows, 
the majority of LASIS sites had at least partially completed the two targeted activities. 



Table 21. SAU’s Progress in Assessing Students with Disabilities/ESL/504 





No action taken 
on this yet 


Planning in 
progress 


Partially 

complete 


Work 

complete 


Assess progress through... 


n 


% 


n 


% 


n 


% 


n 


% 


Assessment with accommodations 


0 


.0% 


2 


15.4% 


9 


69.2% 


2 


15.4% 


Alternative assessment (PAAP) 


1 


7.7% 


2 


15.4% 


5 


38.5% 


5 


38.5% 



Realizing that some SAUs have felt overwhelmed by the magnitude and timeline of 
creating an LAS, information was solicited regarding collaboration between SAUs, the level of 
help needed in specific areas, and the level of confidence in completing the LAS tasks by 2008. 
While 6 of the 13 respondents indicated they were currently collaborating with another SAU for 
the purpose of developing an LAS, an additional 4 respondents indicated they would be 
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interested in collaboration. (Two of the respondents not currently collaborating, did not indicate 
if they would be interested in doing so.) 

Table 22 depicts the perceived level of help needed by SAUs to complete certain tasks. 
LASIS site respondents indicated that the most help was needed in managing the administration 
and scoring of assessment, as well as in developing strategies for using and reporting data. 



Table 22. Level of Help Needed by SAUs to Complete Tasks 





No help 
needed 

1 




2 


Moderate help 
needed 
3 




4 


A lot of help 
needed 
5 




n 


% 


n 


% 


n 


% 


n 


% 


n 


% 


Selecting, developing and adapting 
assessments 


3 


23.1% 


3 


23.1% 


6 


46.2% 


i 


7.7% 


0 


.0% 


Managing the administration and the 
scoring of assessments 


1 


7.7% 


2 


15.4% 


3 


23.1% 


4 


30.8% 


3 


23.1% 


Developing strategies for using and 
reporting data 


0 


.0% 


2 


15.4% 


6 


46.2% 


0 


.0% 


5 


38.5% 


Developing policies 


3 


23.1% 


1 


7.7% 


4 


30.8% 


1 


7.7% 


4 


30.8% 



Using a four-point scale, ranging from do not understand at all (1) to completely 
understand (4), respondents indicated how well students in the Class of 2008 understand the 
graduation requirements that will apply to them. Respondents also indicated how well the 
parents of these same students understand the requirements. Interestingly, Table 23 suggests that 
students may understand slightly more than their parents in this regard. 
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Table 23. Parents’ and Students’ Understanding of Graduation Requirements 





Do not 

understand at all 
1 




2 


3 


Completely 

understand 

4 




n % 


n 


% 


n 


% 


n % 


In your opinion, how well informed are 
parents of incoming ninth grade students 
about the graduation requirements that 
will apply to their children? 


2 16.7% 


5 


41.7% 


4 


33% 


1 8.3% 


In your opinion, how well informed are 
incoming ninth grade students of 
graduation requirements that will apply to 
them? 


1 8.3% 


6 


50.0% 


3 


25% 


2 16.7% 



As with previous LASIS data collection activities, respondents were invited to identify 
boosters and barriers with respect to LAS development/implementation, with the additional 
stipulation that “time” and “money” not be listed. Below is a bulleted list of the perceived 
boosters and barriers. (Parenthetical numbers signify the number of times that a specific booster 
or barrier was listed. Where no number appears, the booster or barrier was identified by only one 
respondent.) 

Boosters 

• LASIS participation (4) 

• Teachers’ attitudes and or professionalism (3) 

• MAP and LAD assessment samples (2) 

• Professional development days that were content specific 

• Networking with other districts 

• Attending State-sponsored workshops and infonnational meetings 

• Sharing of resources: policies, etc. 

• State’s timelines (kept us moving) 

• Board support 

• Various state supports 

• Y ankee ingenuity - “do or die attitude” 

• Other initiatives with mutual goals (e.g., Promising Futures) 

• Meeting with other curriculum coordinators 
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Barriers 

• Deadlines not met by State (5) 

• Too few personnel for LAS work (2) 

• Lack of DOE policy guidance/samples (2) 

• Data-management demands 

• Little capacity for developing replacement assessments and meaningful interventions 

• Teacher burnout 

• Apathy 

• Bureaucratic requirements 

• Changes at the State level 

• Competing priorities (e.g., AYP, highly qualified teachers) 

• Teacher attitudes/skill levels 

• Labor contracts 

• Special education graduation requirements 

• Reliability “issue” 

• Local expertise in assessing and analyzing data 

• Connection of LAS to NCLB 

• Constructing the system as we are incorporating it 

• Ensuring technical rigor 



Using a four-point scale, ranging from not at all confident (1) to very confident (4), 
respondents indicated how confident they were that their SAU would be able to certify that high 
school graduates in 2008 will have met the requirements of the Learning Results in the five 
content areas. (Values of “2” and “3” were left without descriptions.) As of June 2004, one third 
of respondents felt very confident they would be able to certify their graduates of 2008 had met 
the requirements. Table 24 illustrates the results. 



Table 24. SAU’s Confidence Level 




Not at all 
confident 
1 


2 


Very 
confident 
3 4 




n % 


n % 


n % n % 


How confident are you that 
your SAU will be able to 
certify that high school 
graduates in 2008 will have 
met the requirements of the 
Learning Results in the five 
content areas? 


0 .0% 


3 25.0% 


5 42% 4 33.3% 
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PART III: IMPLICATIONS 



The findings in this report reflect the hard work and dedication demonstrated in recent 
years by Maine educators. The 18 SAUs participating in LASIS exemplify the realities of 
Maine’s innovative but complex approach to documenting student achievement of state 
standards. While the vast majority of states have opted for a high stakes standardized testing 
approach to certifying attainment of the standards, Maine has built a system that seeks to 
emphasize educational merit over the efficiencies of large scale standardized testing. 

Throughout the findings of this report it is clear that constructing such a system has required an 
enonnous commitment of time, human resources, and technical expertise — all of which have 
been wedged into already tight schedules and budgets. It has been difficult at times, throughout 
the LAS development process, to remain clear about the broad goal: providing a technically 
sound and student-centered system of assessments to support decisions about progress toward 
and eventual attainment of the standards. As Maine approaches the timeline for the first 
graduating class in 2008, policymakers and educators alike must consider this broader context 
when evaluating the costs and benefits of Maine’s LAS. In the meantime, MDOE will focus 
technical assistance, resources, and programs in response to the findings of this report. 

■ Sharing the Findings 

o Implication: Clearly, one of the most important outcomes of LASIS is the 
capability to share the data with all SAUs in Maine. Valuable insights have 
emerged from the 18 sites regarding “boosters” and “barriers,” summarized in 
the preceding pages, that should help other SAUs refine development 
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strategies, redirect resources, and/or request more targeted technical 
assistance. 

o Action: In addition to mailing the LASIS report to all SAUs, MDOE will 
provide opportunities for additional cross-district and regional workshops to 
support further LAS development based on LASIS findings. The peer-review 
tool and process will be utilized broadly in the coming year in workshop 
settings and in other MDOE supportive work with SAUs. 

■ Gauging Our Progress 

o Implication: While it is clear from the study that capacity is being stretched 
to the limit in many schools and SAUs, it is also clear that enonnous progress 
in LAS development has taken place in the last year. Prom a statewide 
perspective, this bodes well for Maine’s ability to support graduation 
decisions based on student achievement of the Learning Results. 

o Action: MDOE will provide additional technical assistance to support SAUs 
that have not completed the work. Since it is critically important that all 
supportive services and technical assistance efforts by MDOE are coordinated 
and aligned — including school improvement efforts undertaken by the Center 
for Inquiry on Secondary Education (CISE), the federal NCLB school 
improvement team, and the Center for Inquiry on Literacy — MDOE 
leadership will give high priority status in all strategic planning efforts to this 
work. 
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■ Considering the Role of Teachers 

o Implication: The LASIS findings run parallel to the findings of the Task 
Force on Teacher Workload, both of which indicate that the work of 
constructing the LAS has had a significant impact on teachers. The work has 
not only been conceptually challenging but has resulted in widespread 
frustration when in many SAUs neither time nor expertise has been available 
in sufficient quantities to support LAS development. The LASIS findings 
suggest that certain organizational strategies and structures can ameliorate the 
intensity and frustration associated with the work. Where clear, centralized 
organization is present, SAUs appear to have kept energy and commitment 
levels high. 

o Action: MDOE will share the specific learnings from the study and share the 
“boosters” related to teacher role. In addition, Maine teachers surely must be 
among the most assessment literate in the country. MDOE will recognize this 
teacher leadership and use it to help bring other teachers, those newly entering 
the profession, and higher education faculty to the same level. 

■ Considering the Role of Leadership 

o Implication: The study revealed clear characteristics of leadership and 
related organizational practices that are linked with effective LAS 
development. Strong central coordination with clearly delineated 
responsibilities, participation by administrators at both SAU and building 
levels, and the ability to clearly articulate a positive vision for the LAS 
(communicated through a variety of means to all stakeholders) and its link to 
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effective instructional practice — all are characteristics that should be the focus 
of SAUs still struggling to complete LAS development work. In addition, the 
clear expectation that all committee work be reported to administration on a 
regular basis is linked to effective progress. 

o Action: MDOE will include a leadership strand in all LAS regional rounds 

and assessment-related workshops. MDOE will also continue to present at 
state-level association conferences (Maine School Management Association, 
Maine Principals’ Association, and Maine Education Association) to continue 
to build a clearer vision of the role of leadership in LAS development. 

■ Utilizing Time 

o Implication: One of the most frequently mentioned barriers, cutting across 
all the sources of data, is the issue of the lack of sufficient time to both engage 
in LAS development and devote time to other tasks typically associated with 
teacher roles. As noted above, SAUs where effective centralized 
organizational structures exist appear to have fewer issues with teacher time, 
suggesting that careful planning and coordination of work can lead, in many 
cases, to effective use of available time. 

o Action: MDOE will identify and communicate, from the 18 LASIS SAUs as 
well as SAUs statewide, promising practices to maximize effective use of 
available time. A second source of data and possible solutions will likely 
emerge from the Task Force on Teacher Workload, which produced 
preliminary data in the spring of 2004. The Task Force will continue its work 
this fall with a final report expected in January 2005. 
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■ Continuing Need for LASIS 

o Implication: One clear finding from the first year of LASIS is that 

continuing evaluation and data collection will be necessary to detennine the 
impact and effectiveness of the LAS system, particularly in view of the needs 
of the first class to graduate based on standards (2008). 
o Action: The LASIS research team has outlined aspects of the LAS that will 
be under development or implemented over the next four years: scoring and 
reliability aspects of the LAS, using Maine Educational Assessment scores in 
the LAS, evaluating the impact of performance standards approach, and — 
most importantly — evaluating how the system as a whole is impacting all 
students and groups of students (particularly, students with disabilities, 
students in alternative education programs, and students attending the Career 
Technical Education Centers). Thus, MDOE will continue the LASIS project 
at least through the first class to graduate based on standards (2008) and report 
findings and implications each year to policymakers and educators statewide. 
MDOE is defining a research strategy that will redouble its efforts to be a 
more data and evaluation driven organization. This should have the additional 
effect of building capacity in higher education institutions as well as other 
non- and for-profit organizations. 

Of critical importance at this juncture is continuing to build a comprehensive system for 
using assessment data for student and school improvement. Arriving at this point of LAS 
development has clearly required considerable time, effort, and resources. However, now that 
many SAUs are completing the initial phase of work, our attention must turn to putting the data 
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to effective use. In an effort to support local districts in building comprehensive systems of 
accountability (which include data analysis, system-wide planning, student intervention systems, 
and public engagement tools) MDOE will be conducting a four-part series on “Rethinking 
Accountability” during the 2004-05 school year. The LASIS findings will play a useful role in 
district comprehensive planning of an integrated, holistic system of accountability — founded on 
valid and reliable data on student achievement of the Learning Results. The first of the 
Rethinking Accountability workshops was held on August 12 and attracted over 60 SAUs 
represented by over 250 district team members. It is increasingly clear that we must transform 
current systems of curriculum, instruction, professional development, student interventions, and 
public reporting to ensure that the promise of Maine’s Learning Results is attained by all 
students. 

It is also critically important that the context for Maine’s LAS — as supporting evidence 
of student achievement of the Learning Results — be communicated to the wider public. MDOE 
will partner with the Maine Coalition for Excellence in Education (MCEE) in building awareness 
and support for the “mission” of supporting students in meeting the new standards. For all 
audiences — including educators and students — understanding how the LAS is connected to the 
broader context of educational and economic transformation can inspire shared work and a sense 
of purposeful action. 

In the broadest of terms, LASIS was initiated in the context of widespread concerns that 
Maine’s LAS might not be capable or ready to provide valid and reliable data to support 
decision-making about student achievement of the Learning Results. Though work remains, and 
much ongoing evaluation will be required, Maine’s LAS represents one of our nation’s most 
innovative and thoughtful approaches to assessing student learning. We have much to be proud 
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of and much to be grateful for, particularly the incredible effort put forth in recent years by 
teachers, curriculum coordinators, superintendents, and school boards. Their efforts have 
pennitted us to reach this plateau, from which we can see that our destination is attainable. 
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APPENDIX A: INITIAL SURVEY 



District: 



PARTI 



Instructions 

Throughout Part 1 of this survey, you will be asked to describe your district’s level of LAS 
implementation in terms of the five criteria and corresponding rules given in the LAS Guide. 
Because your district may be at different points of development by content area, you will be 
asked to complete this portion of the survey for English language arts, mathematics, science and 
technology, social studies, and health/physical education separately. Each content-area section 
concludes by having you provide an overall rating of your district’s LAS implementation by 
grade span (elementary, middle, and high school). Space for comments is provided throughout. 

We trust that this activity will serve as a valuable self-evaluation for your district and help you 
mark your district’s progress towards full implementation of an LAS. 



In this section of Part 1 of the survey, you are asked to describe LAS implementation in the 
area of English language arts . 
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1 . We begin by asking you to indicate, in the table below, the degree to which your district 
has aligned curriculum and instruction with the Learning Results. For each grade span, 
as well as across all grade spans, please indicate your district’s alignment status by 
entering a “1,” “2,” or “3” separately for curriculum alignment and for instructional 
alignment. 



1 = fully aligned 

2 = partially aligned 

3 = no action taken yet 





English language arts 






K-4 


5-8 


9-12 


K-12 


Alignment of curriculum with the 
Learning Results: 




















Alignment of instruction with the 
Learning Results: 











Comments (use additional sheets if necessary): 
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Criterion 1: Assessment Types - Selection and Distribution 

The term “Assessment Type” is used to classify methods of assessment that measure content and 
skills within a content area at a specified breadth and depth. The distribution of assessment types 
across a content area is the key to determining whether the system of assessment is coherent, and 
whether it is sufficient to support valid and reliable conclusions about student performance in the 
content area as identified by Maine’s Learning Results. Each school administrative unit will 
identify a combination of specific assessments for certification that is representative of the 
character, depth and breadth of the content area. 

RULES 

2. “Assess each content standard for each grade span and content area.'’’’ 



To what extent does your district’s LAS include assessments selected for certification 
purposes that measure all of the content standards? 


Place a S in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


We have selected assessment tasks for each of the content standards. 








We have selected assessment tasks for some of the content standards. 








No action taken yet. 









3. “ Include a minimum of 8-12 assessments for each grade span and content area for 
certification. ” 



Have you have selected/administered enough assessments for certification? 


Place a S in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


More than 12 assessments for each grade span and content area have been 
selected for the LAS 








8-12 assessments have been selected 








7 or fewer assessments have been selected 








No action taken yet. 









4. “ Select a sample of performance indicators based on a method of prioritization (e.g., the 
Department’ s Balance of Representation results). ” 



Please indicate how you have selected a sample of the performance indicators to assess. 


Place a S in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


We relied on the DOE’s Balance of Representation results to establish our 
prioritization and select a sample of performance indicators. 








We relied on our own method to prioritize and select a sample of 
performance indicators. 








No action taken yet. 
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5. “Measure each school level reporting category (content cluster) 5 times. ” 



How many times do you measure each content cluster? 


Place a ^ in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


At least 5 times for each content cluster. 








At least 5 times for some of the content clusters. 








Fewer than 5 times for each of the content clusters. 








No action taken yet. 









6. “Use a variety of assessment types for each school level reporting category (content 
cluster). ” 



Please indicate the extent to which your district uses a variety of assessment types for each 
school-level reporting category (content cluster). 


Place a S i n the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


We have selected assessments representing a variety of types, as described 
by the DOE. 








We have selected assessments representing a variety of types, including 
some not described by the DOE. 








No action taken yet. 









Comments (use additional sheets if necessary): 
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Criterion 2: Comparability 

Comparability means that similar judgments of student perfonnance in the content area (e.g., 
Meets the Standard v. Partially Meets the Standard) reflect similar levels of proficiency in tenns 
of content and skills as identified for Maine’s Learning Results. 

RULES 

7. “ Include a minimum of 75% “common ” assessments ( i. e . , the same assessment is 
administered to each student within the school administrative unit at a time when it is 
instructioncdly appropriate for the individucd student or group of students). ” 



What is the percentage of “common” assessments in your system? 


Place a in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


All of the assessments selected for use in our LAS for certification are 
common. 








At least 75% of the assessments selected are common. 








Fewer than 75% of the assessments selected so far have been identified as 
common. 








No action taken yet. 









8. “ Follow rules for selection and distribution of types when including [non-common] 

classroom assessments (up to 25%) for certification. [Non-common] classroom 
assessments included for certification purposes must meet the requirements for validity 
and reliability as described in Measured Measures. ” 



How are non-common classroom assessments selected for inclusion in the assessment 
system for certification? 


Place a ^ in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


We do not intend to use non-common classroom choice assessments; we 
will only use common assessments. (Skip to next rule.) 








If you intend to use non-common classroom assessments, please answer: 


We have established procedures and have carried out a plan for comparing 
all assessments used for certification (common and non-common 
classroom) to the rules for selection and distribution of assessment types, 
including checking for reliability and validity. 








We have established procedures for comparing non-common classroom 
assessments used for certification to the rules for selection and distribution 
of assessment types, but have not completed the analysis of assessments. 








We have begun to plan and identify procedures for qualifying non- 
common classroom assessments used for certification, but do not yet have 
a complete plan or set of procedures. 








No action taken yet. 
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9. “Measure all students against the same standards, providing appropriate 
accommodations or alternate assessments as necessary. ” 



Does your district provide appropriate accommodations on assessments included in your 
assessment system for certification? 


Place a S in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


We have identified and can provide all necessary accommodations as 
required by students’ IEPs/504 Plans for all assessments used for 
certification in our LAS. 








We have identified all necessary accommodations and can deliver them 
with most assessments. 








We have begun to identify and be able to provide appropriate 
accommodations in our LAS. 








No action taken yet. 









10. “Place scores for each performance indicator on a 4-point scale (1-Does Not Meet, 2- 
Partially Meets, 3-Meets, 4-Exceeds). Scores from assessments that are aligned with 
performance indicators can be converted to a 4-point scale if they have been scored 
using another rubric. ” 



How does your district place scores for each performance indicator on a 4-] 


point sea 


le? 


Place a C in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


All of our assessments are scored directly on a 4-point scale. 








We have established procedures for converting scores for each 
performance indicator to a 4-point scale. 








No action taken yet. 









11. “ Administer and score assessments at each grade span and at each of the following 
levels: classroom, school, school administrative unit; and administer state assessments. 



Please indicate which of the following levels of assessment are administered 


l by you. 


Place a ^ in each box that applies. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


Non-common classroom assessments 








School-wide assessments 








SAU-wide assessments 








MEA 
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12. “Compare results from classroom, common, state and national assessments (when 

available) for consistency - e.g., similar percentages of students achieving at each level 
of proficiency. ” 



To what extent does your district plan and implement procedures to ensure appropriate 
comparability across assessment system components? 


Place a S in the box next to the each phrase that describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


We have a plan to compare results and take appropriate action as 
necessary. 








We are in the process of developing a plan. 








No action taken yet. 









Comments (use additional sheets if necessary): 
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Criterion 3: Replacement 

Replacement is the means by which the assessment system provides the opportunity for a student 
with low performance to demonstrate an acceptable level of proficiency for certification in a 
content area through another assessment. 

RULES 



13. “Provide the opportunity for a student who has not met the standard across the 8-12 
assessments for a content area, at a grade span, to replace low performance on a specific 
assessment with a higher result from another assessment. ” 

“Replace an assessment with another assessment that ensures a variety of types, reflects 
the belief that form must serve function, and maintains distribution across the content 
area ( each content standard must be measured). ” 

“Determine how many times, and when, a student is allowed to replace each assessment 
across the system and under what circumstances, based on the purpose of replacement. ” 



How developed is your district’s policy for replacement assessments? 


Place a S i n the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


We have established a policy for replacement that conforms to the 
requirements of the LAS Guide. 








We are in the process of developing a policy for replacement that 
conforms to the requirements of the LAS Guide. 








No action taken yet. 









Comments (use additional sheets if necessary): 
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Criterion 4: Performance Standards 

Performance standards are the level of student performance across and within the Local 
Assessment System at which it is judged that the student “Meets the Standard” in the content 
area as a whole. 

The LAS Guide provided two possible ways performance standards could be set: a “Mean 
Performance” method and a “Pattern of Performance” method. It is anticipated that few, if any, 
districts will have established student performance standards for the LAS at this point in time, 
because to do so would require they have the complete set of assessments and accompanying 
scored student work. 

RULES 



14. “Apply state -established performance standards based on Criterion 4”. 



Has your district established performance standards? 


Place a S in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


We have selected the “mean performance” method described in the LAS 
Guide. 








We have selected the “pattern of performance” method described in the 
LAS Guide. 








We are still discussing how to proceed in this regard. 
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Criterion 5: Public Reporting 

“Public Reporting” refers to the processes and procedures used to record and aggregate student 
performance information related to the Local Assessment System in order to support annual 
reporting at each grade span for certification of student achievement of the Learning Results and 
school level information about performance at the content cluster level (at a minimum) as 
required by Chapter 127. 

RULES 



15. “Report annually on the percentage of all students at each performance level for the 
[English language arts content area] at the end of each grade span (4 th , 8 th , and 12 th 
grade). ” 

“Report school level information annually for the [English language arts content area ] 
at the end of each grade span (4 th , 8 th , and 12 th grade) at a level of greater specificity 
than content area. At a minimum, the reporting must be at the content cluster level. ” 

“ Aggregate and disaggregate as necessary to report at the two levels noted above 
[content area and content cluster ] and for identified subgroups. ” 

“Ensure that reporting of results for individual students or groups of students maintains 
the confidentiality of individual students. “ 



Please indicate how you annually report the performance results of your local assessment 
system. 


Place a ^ in the box next to the phrase that best describes your current status. 


Grades 

K-4 


Grades 

5-8 


Grades 

9-12 


We report publicly on the results of local assessments. 








We are developing strategies for reporting publicly on the results of local 
assessments. 








No action taken yet. 









Comments (use additional sheets if necessary): 
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Overall Rating (English language arts): 

Local Assessment System Development and Implementation 

Using the three continua on the following page, please indicate where your district is in terms 

of implementing a LAS. Use the description that most applies to your district’s understanding, 

support for a LAS, implementation, and evidence : 



a. Exploring - Becoming acquainted with the ideas. Still building support for an LAS 
within the district and schools. Not very familiar with the LAS Guide. Still in the early 
planning stages. Documentation may be available to some, but not in a consistent fonn 
or quality for public reporting or peer review. 

b. Transitioning (early) -Have good beginning with assessment literacy and have good 
capacity (many people) to build towards a comprehensive LAS. Key persons in district 
and schools have a working knowledge of most of the LAS Guide. Have strong 
support for the LAS in key parts of the district/schools. Have several parts of an LAS 
in place, but we are working on getting alignment and agreement across schools, 
grades, teachers, and/or content areas. Documenting our process as we go. 

c. Transitioning (more advanced) - Have district LAS policies and procedures in place. 
Many people (administrators, faculty, students, parents) understand and implement 
system well. The value of the LAS is reflected in such things as regular reporting to 
policy makers and stakeholders, integrated use in planning, and in discussions of 
allocation of resources. Have established processes to review the system components, 
and have documented evidence of how system meets principles of coherence, 
sufficiency, and fairness. 

d. Transforming - Many teachers can extend the LAS beyond district and state 
requirements. District has fine-tuned LAS requirements to fit its individual 
circumstances, while meeting core principles of coherence, sufficiency, and fairness. 
LAS results are incorporated regularly into district and school decisions as valuable 
data. District has a thorough process and regularly reviews evidence to certify and 
improve its LAS. 
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Given the descriptions on the previous page, place an “X” on the line according to your district’s 
current phase of development. 



English Language Arts 



Grade Span: K-4 

1 








1 

Exploring 


Transitioning 

(early) 


Transitioning 

(advanced) 


w 

Transformin 


Grade Span: 5-8 






1 


Exploring 


Transitioning 

(early) 


Transitioning 

(advanced) 


1 

Transformin 


Grade Span: 9-12 






| 


Exploring 


Transitioning 

(early) 


Transitioning 

(advanced) 


1 

Transformin 
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APPENDIX B: PHONE INTERVIEW PROTOCOL 

5 December 2003 

To: Wayne, Sara, Owen 

From: Ted 

Re: Addressing the matter of tape recording the phone interview 



Please carefully follow the tape-recording script below. You need not read this script verbatim, 
mind you — although I tried to write it so that you comfortably could. 



First things first 

Begin by describing the purpose and nature of this first phone interview: 

• The purpose of this call is to obtain additional information and insights regarding local 
assessment system development. 

• At the end of the interview, logistics for the site visit will be briefly discussed. 

• The call should last no more than an hour. 



The script regarding tape recording 

With your permission , I’d like to tape record our conversation so I can be sure to have an 
accurate record of the information you provide me. Before you tell me whether this is ok, let me 
assure you of several things: 

• Only I will listen to your tape, and only for the purpose of taking detailed notes. (This 
will allow me to be a more attentive conversationalist today!) 

• At any time during our conversation, you may ask that I turn off the tape recorder — 
whether for a particular comment you wish to make or for the remainder of the interview. 

• After I take notes from your tape, the tape will be stored in locked file drawer in the office 
of Dr. Walter Harris, Director of the University of Maine Center for Research and 
Evaluation. Your tape will be destroyed at the completion of the implementation study 
(summer 2004), and the only reason I would listen to it before then would be to clarify my 
notes. 

So, unless you have any questions that I may answer, do I have your permission to tape record 
this interview? 
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The interview questions 

For your convenience, I’ve reproduced the interview questions that Jill and I wrote and 
distributed on 12-3-04. [As you conduct each interview, keep in mind the overall goals: (a) 
eliciting the “how” and “why” for the dimensions below and (b) eliciting perceptions regarding 
“boosters” and “barriers” for these dimensions. Please probe as necessary.] 



1) How does your SAU manage/coordinate the work plan and timelines for local assessment 
system development? 

a) Who is providing leadership for this work? 

b) What structures are in place (e.g., committees, work groups)? 

c) What strategies is your SAU using to provide time for this work? 

2) How would you describe the attitude toward the local assessment system requirement in your 
SAU? 

a) How have SAU leaders described the purpose, rationale, utility, etc. of your work toward 
meeting this requirement? 

b) How do teachers in your SAU feel about this work? 

c) To what extent are parents, school board members, and other community members aware 
of this work? How do they feel about it? 

3) To what extent has your SAU used DOE resources for local assessment system development 
(e.g., LAS Guide, Measured Measures, LAD & MAP websites, regional professional 
development)? 

4) To what extent has your SAU developed/modified policies in response to local assessment 
system requirements (e.g., adopting a standards-based report card; developing interventions 
for students not meeting performance standards)? 

5) What efforts has your SAU made to include all students in the local assessment system? 

a) What progress is being made in providing accommodations and alternate assessments, as 
appropriate? 

b) What efforts are being made to include students in vocational and alternative education 
programs? 
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6) The DOE is planning another meeting for LASIS participants to share their experiences with 
local assessment system development. 

a) Would you and your team be likely to attend such a meeting? 

b) “Sharing groups” will be structured around common roles and issues. What role-alike 
groups would be helpful to members of your team? (For example, would teachers like to 
meet by grade span or discipline?) What issues would you and your team like to discuss 
with people from other sites (e.g., scoring and benchmarking student work)? 

7) [Interviewer: ask contact person to schedule meetings with representatives from each content 
area during the site visit.] 
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APPENDIX C: SITE VISIT PROTOCOL 



LASIS Site Visit Protocol & Interviews 



Pre-Site Visit Preparation: 

Questions below may already have been answered during the phone-interview. It will be 
important to go over phone-interview reports and extract responses that fit the site-visit 
questions. 

It will also be important to review the information from Nov. 7 survey’s (for probing purposes or 
to follow-up on striking features of the initial narratives). 

The overall goals of the site visit include: 

• Validating information already obtained about the implementation of local assessment 
systems from surveys and telephone interviews, 

• Updating previously obtained information, and 

• Gaining new information that the structure of previous interviews and surveys may have 
obscured. 

• Keep in mind we’re looking at all components of the L.A.S. (i.e. all assessments 
including sub-set for certification) 

Collection of Artifacts: 

If districts have artifacts that pertain to the L.A.S. design and implementation (i.e. policies, 
structural designs, standard based report cards, newsletters, etc.) please ask for an electronic 
version to be sent to you. All documents should then be forwarded to Walt. 

Describing the Purpose of the Site Visit Interview 

Because you will be interviewing new people throughout your visit it will be important to 
describe the purpose and nature of the interview: 

The purpose of the interview is to obtain additional information and insights regarding the 
current status of local assessment system development. As we go through the interview, please 
feel free to add any additional information that you feel would be helpful to this study. 
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Suggested script regarding tape recording: 

It will be important to read/highlight this script to each person who will be interviewed. 

Teachers may feel uncomfortable being interviewed and taped. Assure them that you won’t tape 
if they’re not comfortable or if they want to share infonnation that they don’t want taped. 

With your permission , I’d like to tape record our conversation so 1 can be sure to have an 
accurate record of the information you provide me. Before you tell me whether this is ok , let me 
assure you of several things: 

• Only I will listen to your tape, and only for the purpose of taking detailed notes. (This 
will allow me to be a more attentive conversationalist today!) 

• At any time during our conversation, you may ask that I turn off the tape recorder — 
whether for a particular comment you wish to make or for the remainder of the interview. 

• After I take notes from your tape, the tape will be stored in locked file drawer in the office 
of Dr. Walter Harris, Director of the University of Maine Center for Research and 
Evaluation. Your tape will be destroyed at the completion of the implementation study 
(summer 2004), and the only reason I would listen to it before then would be to clarify my 
notes. 

So, unless you have any questions that I may answer, do I have your permission to tape record 
this interview? 

Audience: Identified leadership team and/or any other district leaders that seem appropriate or 
are recommended. . . 

Questions 

1. SAU Organizational Structure, Time and Funds: 

Structure: 

a. What does the structure look like? 

b. Who’s providing the leadership? How many people are involved in the work? Please 
provide titles/roles and name of the leader. 

c. How is the L.A.S. work communicated to K-12 staffs that have to implement the 
system but are not involved in the leadership structure? 



Time: 

d. How much time is being spent on the LAS work (early release, late start, after school, 
etc.)?, How often do groups meet? Are you spending too little time? Too much? What 
is the impact on other work and/or priorities? 

Funding: 

e. How are leaders / teacher leaders/ and all teachers working on LAS being compensated 
for the work? (Where are the funds coming from? How are people paid for their 
involvement in this work?) 
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Timeline of Curriculum/As sessment/Instruction Process: 

f. Describe the duration of effort (LAS vs. Curriculum)? It is important to clarify the 
difference between the work of aligning the curriculum to Maine’s Learning Results and 
the work of designing and implementing a local assessment system using the LAS Guide. 

4. Vision 

a. You know, there are a variety of beliefs about the LAS, the requirements and how it 
will impact teachers, students and learning. What do you think about this? Explain and 
tell us why 

5. Disposition & Attitudes 

a. How would you describe the attitude toward the local assessment system requirement 
in your district? 

b. Does the current attitude reflect a change in attitude over the last several years? If so, 
how would you describe this change? 

c. How do teachers in your district feel about this work? 

d. To what extent are parents and other community members aware of this work? How 
do they feel about it? 

e. Can you provide evidence of attitudes in your SAU? 

6. Communication with Students, Parents, Community & School Board 

a. How are you communicating with each group? 

b. What are you communicating? 

c. What are the responses & reactions from these groups? (Do you feel they are gaining 
an understanding of the impact of this work? What are they doing to show you this 
understanding ? ) 

d. What are your greatest needs in reaching out to the community and parents? 

7. Resources: 

a. How do you determine what resources to use when designing and implementing the 
L.A.S.? 

b. Within the L.A.S. Guide, what clarification or elaboration would you find most 
helpful? 

c. Other DOE resources. . .(*see web list) 

c. Which non DOE resources are you using? 

d. If so, how have they been useful? 

8. Policy: (update previous responses and provide examples of drafts/final copies) 

a. To what extent has your district drafted/developed/modified policy in response to local 
assessment system requirements (e.g. adopting a standards-based report card, 



Local Assessment System Implementation Study ( 2003-2004 ): Technical Report 

August 2004 




82 



replacement, developing interventions for students not meeting performance standards, 

promotion, graduation requirements)? 

b. Do you have any written approved “policies”? 

9. “All Students” 

a. What efforts has your district made to include all students in the local assessment 
system? 

b. What progress is being made in providing accommodations and alternate assessments, 
as appropriate? 

c. What efforts are being made to include students in vocational programs, alternative 
education programs and adult education high diploma requirements? (separate responses 
by each category) 

10. Common Assessment Implementation 

a. How much time is used to schedule, administer and score common assessments? What 
are some boosters and barriers in this area? 

1 1 . Additional Infonnation 

a. What is the status of your implementation of Career Preparation, Modem and Classical 
Languages, and Visual and Perfonning Arts? Please explain 

b. With respect to LAS design, implementation and progress/future plans, is there anything 
else you wish to share that we haven’t touched on today? 

Overall: What is helping or hindering your progress towards an LAS? 
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APPENDIX D: LAS CHECKLIST 



Based on the LAS Guide: 

Principles and Criteria for the Adoption of Local Assessment Systems 

Principles 

Coherence 

Fairness 

□ The PK - 4 curriculum document for 5 content areas with demonstrated alignment 
with Maine’s Learning Results 

□ The 5-8 curriculum document for 5 content areas with demonstrated alignment with 
Maine’s Learning Results 

□ The 9-12 Programs of Study document for the 5 content areas with demonstrated 
alignment with Maine’s Learning Results 

□ Communication guidance on “pathways” for students 

□ Five sample assessments and curriculum and instruction artifacts. These represent 
each content area and grade span. 

Principle and Criterion 1 



Sufficiency 

Assessment Types - Selection and Distribution 

□ A list of prioritized perfonnance indicators for each content area and grade span with 
an explanation of method used if not DOE Balance of Representation 

□ English language arts templates 
PK - 4 

5-8 

9-12 

□ Health and Physical Education templates 
PK - 4 

5-8 

9-12 

□ Mathematics templates 
PK - 4 

5-8 

9-12 
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□ Science and Technology templates 
PK-4 

5-8 

9-12 

□ Social Studies templates 
PK-4 

5-8 

9-12 



Criterion 2 

Comparability 

□ Common / classroom assessments labeled on templates 

□ Policy documents related to the participation of all students (accommodations and 
alternate) 

□ 2 sample IEPs that represent the range of assessment participation (names removed) 

□ Conversion to 4 point scale protocols (if applicable) 

Criterion 3 

Re placement 

□ Replacement Policy including remediation /intervention plans if available. 

Criterion 4 



Performance Standards 

□ Performance Standards Set selected 

Criterion 5 



Public Reporting 

□ Public Reporting Policy 

□ Sample Reporting Fonnat 
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□ Professional Development Plan 

□ Communication Strategy 

□ Data Management Plan 

□ Implementation of Career Preparation, Modem and Classical Languages, and Visual 
and Perfonning Arts Plan 

□ Other Plans... 
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APPENDIX E: PEER REVIEW INSTURMENT 



LAS 

Principles 



Coherence 

Fairness 

(Opportunity to 
learn) 



CONT. 



Peer 

Reviewer 

Question 

What steps 
have been 
taken to 
ensure that 
ail students 
have the 
opportunity to 
learn, 

develop and 
demonstrate 
the 

knowledge 
and skills in 
Maine’s 
Learning 
Results ? 

(All = regular 

education, 

special 

education, 

vocational 

education, 

alternative 

education ...) 



Source of Evidence 



Curriculum 

documents 

Course of study 
documents 



1 

The evidence does 
not demonstrate 
accomplishment of 
any aspect of the 
principle/criterion. 



• Communication 
guidance on 
pathways to 
ensure 

opportunity to 

I earn 

Evidence of 
Accomplishment 

The PK-4 — 5-8 List supporting evidence: 

curriculum sequence 

and the 9-12 programs 

of study for each of the 5 

disciplines include 

courses/ course 

sequencing that 

correspond to the List missing evidence: 

content standards of the 

Learning Results to 

provide the opportunity 

for all students to learn 

the knowledge and skills 

in the Learning Results. 



Extent to Which Evidence is Present 



ind 


licate your general ral 


:ing: 


2 


3 


4 


5 




The evidence 
demonstrates 
accomplishment of 
some, but not all, 
aspects of the 
principle/criterion. 




The evidence clearly 
demonstrates 
accomplishment of all 
aspects of the 
principle/criterion. 



Curriculum documents aligned with MLR? 



Content Area 


PK-4 


5-8 


9-12 


ELA 








Health/Physical 

Education 








Mathematics 








Science & 
Technology 








Social Studies 
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Peer 

Reviewer 

Question 



Source of Evidence 



5 sample 
assessments 
(including locally 
developed or 
adapted 

assessments) from 
across 5 content 
areas and 3 grade 
spans; curriculum 
and instruction 
artifacts used to 
develop skills and 
knowledge for each 
assessment 



The evidence does 
not demonstrate 
accomplishment of 
any aspect of the 
principle/criterion. 



Evidence of 
Accomplishment 

The 5 sample List supporting evidence: 

assessments with the 

artifacts of curriculum 

and instruction represent 

an adequate opportunity 

for students to 

demonstrate the 

knowledge and skills in List missing evidence: 
the Learning Results. 
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LAS 

Principle & 
Criterion 1 
Sufficiency 

Assessment 

Type- 

Selection 

and 

Distribution 



Peer 

Reviewer 

Question 

Has the SAU 
selected a 
collection of 
assessments 
to meet the 
rules for this 
principle/ 
criterion? 



Source of Evidence 



• Description of 
process (i.e. DOE 
Balance of 
Representation 
or local method) 

• List of prioritized 
performance 
indicators at each 
grade span for 
each of the 5 
content areas. 

Evidence of 
Accomplishment 
Description of a 
process to prioritize 
performance 
indicators; a list of 
indicators for each 
grade span for each of 
the 5 content areas. 



1 

The evidence does 
not demonstrate 
accomplishment of 
any aspect of the 
principle/criterion. 



List supporting evidence: 



Extent to Which Evidence is Present 



indicate your general rating: 



2 


3 


4 


5 




The evidence 
demonstrates 
accomplishment of 
some, but not all, 
aspects of the 
principle/criterion. 




The evidence clearly 
demonstrates 
accomplishment of all 
aspects of the 
principle/criterion. 



Content Area 


PK-4 


5-8 


9-12 


ELA 








Health/Physical 

Education 








Mathematics 








Science & 
Technology 








Social Studies 









(LAS Guide, Pg. 9) 



List missing evidence: 



CONT. 
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LAS Peer Source of Evidence 

Principle & Reviewer 

Criterion 1 Question 

CONT. • Content Area / Grade 

Span Templates (15) 



Evidence of 
Accomplishment 
A collection of at least 8 
assessments have been 
identified for each content 
area and includes: 

• At least one measure of 
each content standard. 

• At least 5 measures for 
each content cluster. 

• A variety of 
assessment types for 
each school level 
reporting category. 
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89 




August 2004 





90 



Criterion 2 



Comparability 



Peer 

Reviewer 

Question 

Has the SAU 
ensured that 
similar 

judgments of 
student 
performance 
in a content 
area reflect 
similar levels 
of proficiency 
in terms of the 
content and 
skills as 
identified in 
Maine’s 
Learning 
Results ? 



Source of Evidence 



Extent to Which Evidence is Present 






Templates with 



1 2 



common 



The evidence 



assessments 
labeled for the 5 
content areas and 
3 grade spans 



does not 
demonstrate 
accomplishm 
ent of any 
aspect of the 
principle/crite 



rion. | 

Evidence of 
Accomplishment 

At least 75% of each List supporting evidence: 

collection of 

assessments used for 

certification are 

common assessments. 



indicate your general rating: 



3 


4 


5 


The evidence 
demonstrates 
accomplishment of 
some, but not all, 
aspects of the 
principle/criterion. 




The evidence clearly 
demonstrates 
accomplishment of all 
aspects of the 
principle/criterion. 



List missing evidence: 



CONT. 
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Criterion 2 


Peer 

Reviewer 

Question 


Source of Evidence 


Extent to Which Evidence is Present 

indicate your general rating: 






• Policy documents 


1 


2 


3 


4 


5 


CONT. 




for alternate 
assessment 
/accommodations 
and sample 
assessments. 

• 2 sample lEPs 
(names removed) 


The evidence 
does not 
demonstrate 
accomplishment 
of any aspect of 
the 

principle/criterion. 




The evidence 
demonstrates 
accomplishment of 
some, but not all, 
aspects of the 
principle/criterion. 




The evidence clearly 
demonstrates 
accomplishment of all 
aspects of the 
principle/criterion. 






Evidence of 
Accomplishment 
















Measure all students 
against the same 
standards, providing 
appropriate 
accommodations or 
alternate assessments 
as necessary. 


List supporting evidence: 












(LAS Guide, Pg. 15) 


List missing evidence: 
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Criterion 2 



Peer 

Reviewer 

Question 



Source of Evidence 



Extent to Which Evidence is Present 



Indicate your general 



rating: 



CONT. 



• Sample 
assessments (with 
scoring guides if 
used) 

• Conversion 
protocol 



1 



The evidence 
does not 
demonstrate 
accomplishm 
ent of any 
aspect of the 
principle/crite 
rion. 



The evidence 
demonstrates 
accomplishment of 
some, but not all, 
aspects of the 
principle/criterion. 



The evidence 
clearly 

demonstrates 
accomplishment 
of all aspects of 
the 

principle/criterio 

n. 



Evidence of 
Accomplishment 



All scores for each 
performance indicator 
will be placed on a 4- 
point scale. 

(LAS Guide, Pg 15) 



List supporting evidence: 



List missing evidence: 
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Criterion 3 



Replacement 



Peer 

Reviewer 

Question 

Is there an 
SAU 

replacement 
policy 
allowing all 
students to 
replace low 
performance 
on an 

assessment 
with better 
performance 
on another 
assessment 
(within the 
set of 

assessments 

for 

certification?) 



Source of Evidence 



Replacement Policy 



Evidence of 

Accomplishment 

A replacement policy is in place at 

each grade span and for each of the 

5 content areas that includes: 

• The opportunity for a student to 
replace a low performance with a 
better performance across the 
collection of assessments for 
each content area. 

• A provision that replacements 
ensure a variety of types, the 
belief that form follows function, 
and maintains the distribution 
across the content area. 

• How many times, when, and 
under what circumstances a 
student is allowed to replace 
based on the Purpose of 
Replacement. 

• The opportunity for additional 
instruction/remediation required 
before the replacement. 

(LAS Guide, Pg. 18) 
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2004 





Criterion 4 



Source of Evidence 



Peer 

Reviewer 
Question 

Has the SAU • Performance 
Performance made a Standards Set/ Policy 

Standards decision Document 

about the 
performance 
standards to 

be applied? 

Evidence of 

Accomplishment 

A set of performance 
standards to apply for 
certification purposes has 
been identified by the 
SAU. 

(LAS Guide, Pg. 4) 



Local Assessment 
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Criterion 5 



Peer 

Reviewer 

Question 



Source of Evidence 



Extent to Which Evidence is Present 



indicate your general rating: 



Public 

Reporting 



Is the SAU 
prepared to 
report the 
performance 
levels of its 
students in 
the 5 content 
areas at the 
3 grade 
spans? 



Policy for Reporting 
Sample Reporting 
Format 



1 



The evidence does not 
demonstrate 
accomplishment of any 
aspect of the 
principle/criterion. 



The evidence 
demonstrates 
accomplishment of 
some, but not all, 
aspects of the 
principle/criterion. 



The evidence 
clearly 

demonstrates 
accomplishment 
of all aspects of 
the 

principle/criterion. 



Evidence of 
Accomplishment 



The SAU is prepared to 
report to the public annually: 

• the percentage of 
students in each of 
the 4 performance 
levels in each 
content area at each 
grade span 

• school level 

information for the 5 
content areas at the 
end of each grade 
span at a greater 
specificity than the 
content area 

(LAS Guide, Pg. 25) 



List supporting evidence: 



List missing evidence: 
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Peer 

Reviewer 

Question 

Has the SAU 
developed the 
policies/plans 
for local 
assessment 
systems listed 
in Chapter 127 
Instructional 
Program, 
Assessment, 
and Diploma 
Requirements- 
Section 4.02? 



Source of Evidence 



Graduation Policy 
Policy regarding role of MEA 
Professional Development 
Plan 

Communication Strategy Plan 
Data Management Plan 
Implementation of Career 
Preparation, Modern and 
Classical Languages, and 
Visual and Performing Arts 
Plan 



Evidence of 

Accomplishment 

Policies and plans required by 
Chapter 127 Instructional Program, 
Assessment, and Diploma 
Requirements 



Local Assessment 



August 2004 



Extent to Which Evidence is Present 

indicate your general rating: 

4 5 

The evidence 
clearly 

demonstrates 
accomplishment of 
all aspects of the 
principle/criterion. 



List supporting evidence: 



List missing evidence: 



The evidence 
does not 
demonstrate 
accomplishment 
of any aspect of 
the 

principle/criterion. 



The evidence 
demonstrates 
accomplishment 
of some, but not 
all, aspects of the 
principle/criterion 
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APPENDIX F: FEEDBACK FORM 



Peer Reviewer Feedback 
LASIS May 18, 2004 

The intent of this form is to gather your feedback on the training and the materials used during 
these two days. We hope to use a similar process and similar materials in regional seminars 
around the state. Please offer specific comments whenever possible. 

1. Were the AM training and materials helpful with your use of the review tool? 

Yes Somewhat No 

Comments: 



2. Did it make sense to have the Peer Review tool organized by principles and criteria as is done 
in the LAS Guide? 



Yes 



Somewhat 



No 



Comments: 



3. Were the guiding Peer Reviewer Questions clear? 

Yes Somewhat No 



Comments: 



4. Did the evidence column clarify what you needed to document in order to complete your 
rating? 



Yes 



Somewhat No 



Comments: 
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5. Was the use of the 1-5 rating scale helpful for the task at hand? 

Yes Somewhat No 

Comments: 

6. Was the supporting and missing evidence appropriate feedback? 

Yes Somewhat No 

Comments: 

7. Was the process of scoring individually and then coming to consensus effective and efficient? 

Yes Somewhat No 

Comments: 

8. Did the process allow you to identify accomplishments and next steps for the SAUs you 
reviewed? 

Yes Somewhat No 

Comments: 

I. Will the information you have received for your SAU help you move forward? 

Yes Somewhat No 

10. What else could a process like this provide? 

I I . How was participating in this process useful to you? 
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APPENDIX G: JUNE 2004 SURVEY 



School Administrative Unit Progress on the 
Implementation of a Local Assessment System 

ANSWERS SHOULD PERTAIN TO ENTIRE SAU (SCHOOL DISTRICT) 

1. In what superintendent region is your school administrative unit (SAU)? 



Aroostook County 
Cumberland County 
Hancock County 
Penquis 

2. Our SAU is a (an) . 
private school 
MSAD 
Union 



Midcoast 
Kennebec Valley 
Washington County 
Western Maine 

town with individual 
CSD 

Unorganized territory 



York 

Unorganized territories 

SEE MAP ON LAST PAGE 

supervision 



3. What grade levels are included in your SAU? (Circle all that apply.) 
PreK — K — 1 — 2 — 3 — 4 — 5 — 6-7-8 — 9 — 10 — 11 — 12 



4 . How many students are enrolled in your SAU? 

Fewer than 200 851-2000 

201-400 More than 2000 

401-850 



5. Rate the extent to which each of the following is involved in the 

development and implementation of your local assessment system (LAS) . 

If this role or structure does not exist in your SAU, please indicate in 
the box provided. 





This role or 
structure does 
not exist 

(please check) 


No 

involvement 

1 


2 


3 


Extensive 

involvement 

4 


ROLE: 












Assessment specialist 












Assistant superintendent 












Communications personnel 












Content leader 












Curriculum coordinator 












Data management specialist 












Principal 












Superintendent 












Other role : 












STRUCTURE : 












Curriculum committee 
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K-12 content area team 












Local assessment team 












Other structure: 













6. Rate your SAU' s progress on each of the following activities. Your rating 
should represent the progress being made in the specified content area 
across each grade span. 



ENGLISH/LANGUAGE ARTS 


1 = No action taken on this yet 

2 = Planning in progress 

3 = Partially complete 

4 = Work complete 


Our SAU has . . . 


K-4 




5-8 




9-12 


1 


2 


3 


4 




1 


2 


3 


4 




1 


2 


3 


4 


a. developed curriculum documents aligned 
with Maine Learning Results. 






























b. aligned instruction with curriculum and 
Maine Learning Results. 






























c. selected a sampling of the performance 
indicators to assess using Maine's 
Balance of Representation or some other 
method of prioritization. 




















; 










d. identified at least 8 assessments for 
each grade span that : 

• samples every content standard 

• measures each content cluster five times 

• provides a variety of assessment types by 

clusters 

• identifies 75% of assessments as common 






























e. provided professional development in 
common scoring for teachers. 






























Our SAU is . . . 






























f. prepared to fully implement the 

expectations of the LAS in the fall of 
2004 (refer to informational letter #117) 































MATHEMATICS 


1 = No action taken on this yet 

2 = Planning in progress 

3 = Partially complete 

4 = Work complete 


Our SAU has . . . 


K-4 




5-8 i 


; 9-12 | 


1 


2 


3 


4 


0 


1 


2 


3 


4 


1 


2 


3 


4 


a. developed curriculum documents aligned 
with Maine Learning Results. 










H 


















b. aligned instruction with curriculum and 
Maine Learning Results. 




























c. selected a sampling of the performance 
indicators to assess using Maine's 
Balance of Representation or some other 










Q 
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method of prioritization. 


























d. identified at least 8 assessments for 
each grade span that : 

• samples every content standard 

• measures each content cluster five times 

• provides a variety of assessment types by 

clusters 

• identifies 75% of assessments as common 


























e. provided professional development in 
common scoring for teachers. 


























Our SAU is . . . 


























f. prepared to fully implement the 

expectations of the LAS in the fall of 
2004 (refer to informational letter #117) 



























SCIENCE AND TECHNOLOGY 


1 = No action taken on this yet 

2 = Planning in progress 

3 = Partially complete 

4 = Work complete 


Our SAU has . . . 


K-4 ! 


J 5-8 1 


; 9-12 i 


1 


2 


3 


4 


1 


2 


3 


4 


1 


2 


3 


4 


a. developed curriculum documents aligned 
with Maine Learning Results. 


























b. aligned instruction with curriculum and 
Maine Learning Results. 


























c. selected a sampling of the performance 
indicators to assess using Maine's 
Balance of Representation or some other 
method of prioritization. 


























d. identified at least 8 assessments for 
each grade span that : 

• samples every content standard 

• measures each content cluster five times 

• provides a variety of assessment types by 

clusters 

• identifies 75% of assessments as common 


























e. provided professional development in 
common scoring for teachers. 


























Our SAU is . . . 


























f. prepared to fully implement the 

expectations of the LAS in the fall of 
2004 (refer to informational letter #117) 



























SOCIAL STUDIES 


1 = No action taken on this yet 

2 = Planning in progress 

3 = Partially complete 

4 = Work complete 


Our SAU has . . . 


K-4 I 


1 5-8 L 


! 9-12 1 


1 


2 


3 


4 


1 


2 


3 


4 


1 


2 


3 


4 


a. developed curriculum documents aligned 
with Maine Learning Results. 


























b. aligned instruction with curriculum and 
Maine Learning Results. 


























c. selected a sampling of the performance 
indicators to assess using Maine's 
Balance of Representation or some other 
method of prioritization. 


























d. identified at least 8 assessments for 
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each grade span that : 

• samples every content standard 

• measures each content cluster five times 

• provides a variety of assessment types by 

clusters 

• identifies 75% of assessments as common 


























e. provided professional development in 
common scoring for teachers. 


























Our SAU is . . . 


























f. prepared to fully implement the 

expectations of the LAS in the fall of 
2004 (refer to informational letter #117) 



























HEALTH AND PHYSICAL 
EDUCATION 


1 = No action taken on this yet 

2 = Planning in progress 

3 = Partially complete 

4 = Work complete 


Our SAU has . . . 


K-4 ; 


1 5-8 1 


! 9-12 | 


1 


2 


3 






2 


3 


4 


1 


2 


3 


4 


a. developed curriculum documents aligned 
with Maine Learning Results. 


























b. aligned instruction with curriculum and 
Maine Learning Results. 


























c. selected a sampling of the performance 
indicators to assess using Maine's 
Balance of Representation or some other 
method of prioritization. 


























d. identified at least 8 assessments for 
each grade span that : 

• samples every content standard 

• measures each content cluster five times 

• provides a variety of assessment types by 

clusters 

• identifies 75% of assessments as common 


























e. provided professional development in 
common scoring for teachers. 










■ 
















Our SAU is . . . 


























f. prepared to fully implement the 

expectations of the LAS in the fall of 
2004 (refer to informational letter #117) 










r 






□ 


r 









7. Reflecting upon the above activities (a through f collectively) , 

indicate the progress your SAU has made toward developing an LAS in 
Modern and Classical Languages, Visual and Performing Arts, and Career 
Preparation (which have a different timeline for completion) . 



1 = No action taken on this yet 

2 = Planning in progress 

3 = Partially complete 

4 = Work complete 





K-4 




5-8 i 




9-12 | 


1 


2 


3 


4 




1 


2 


3 


4 




1 


2 


3 


4 


Modern and Classical Languages 




























Visual and Performing Arts 
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Career Preparation 



8 . Please indicate the progress your SAU has made on each of the following 
activities toward implementing an LAS using the scale provided. 



Our SAU has . . . 


K- 12 | 


No action 
taken on 
this yet 
1 


Planning 

in 

progress 

2 


Partially 

complete 

3 


Work 

complete 

4 


a. developed a policy for replacement assessments. 










b. selected either the recommended or alternative set 
of performance standards from the LAS quide . 










c. developed a procedure for public reporting of 

scores at the required levels for certification. 











9 . 



Is your SAU currently collaborating with one or more SAUs for the 
purpose of developing an LAS? 

• Yes • No 



10. If you answered no to the previous question, would your SAU be 
interested in collaboration? Yes No 



11. To what extent is your SAU prepared to assess the progress of students 
with disabilities/ESL/504 toward achieving the Learning Results through 





No action 
taken on 
this yet 
1 


Planning 

in 

progress 

2 


Partially 

complete 

3 


Work 

complete 

4 


Don 7 1 
know 

5 


Assessment with accommodations 












Alternate assessment (personalized 
alternate assessment portfolio PAAP) 













12. For each of the following, indicate the level of help needed by your 
SAU . 





No help 
needed 

1 


2 


Moderate 

help 

needed 

3 


4 


A lot of 
help 
needed 

5 


Selecting, developing and adapting 
assessments 












Managing the administration and the 
scoring of assessments 












Developing strategies for using and 
reporting data 












Developing policies (e.g., graduation, 
replacements, accommodations) 















No action 
taken on 
this yet 


Planning 

in 

progress 


Partially 

complete 


Work 

complete 




i 


2 


3 


4 
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13 . To what degree has your SAU developed a 
comprehensive scoring system that will 
meet technical standards? 













Not at 
all 

confident 






Very- 

confident 




1 


2 


3 


4 


14 . How confident are you that your SAU will 
be able to certify that high school 
graduates in 2008 will have met the 
requirements of the Learning Results in 
the five content areas? 
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Do not 
understand 
at all 






Completely 

understand 




1 


2 


3 


4 


15. In your opinion, how well do the parents 
of incoming ninth grade students (Class 
of 2008) understand the graduation 
requirements that will apply to their 
children? 










16. In your opinion, how well do the incoming 
ninth grade students (Class of 2008) 
understand the graduation requirements 
that will apply to them? 











17. We know that "time" and "money" can be both boosters and barriers to the 
development/implementation of an LAS. Please help us identify other 
boosters and barriers to the LAS process that have facilitated or impeded 
your progress. 



Boosters 




Barriers 


l . 


1 . 




2 . 


2 . 





Maine Superintendent Regions 
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